refactoring and spaghetti code

from engineering experience:

- Using map instead of repeating calls for function with more than two parameters looks bad and unmaintainable for number of calls 5 or less.

- Using dynamic SQL for creation of similar tables , describing same parts in while loop looks bad if number of tables less than 4.

Of cause you have not use this tech's if the code is evolving and probably would be changed (or it is permanent solution).

refactoring

Appeared, that I have no suitable refactoring tool.

Started refactoring utility.
Platform is C, prototype in perl + postgres.

Perl appears great tool for prototyping of such tool(actually specific processor of set of files).
It provides you working prototype in times faster that it can be done in C.

First problems:

- Lack of time. I have only time when travelling between home and job, It takes about 1.5 hours, but not all time can be used for code input. That way version 1.0 only for Linux can take up to next new year. Actually it is possible that it would be never fixed because of changes in schedules, regime and load on paid job.

- Problem of platform choice. It have to be C because it is probably would be utility for GNU/Linux. Checked standards and appeared that C standard changed from last time I used pure C, but not C++. There is now C99 standard, and there is also draft standard with changes dated by this year.Also in GNU/Linux appeared used non standard GNU extensions of C. Decided to use C90, that way, because it is most commonly supported.

- Data structures. It is a lot of choices with pro's and con's. Decided to play with Perl prototype first. Using DS simple in implementation, with ability to change this to more fast but complex.

- Design questions. (sizes of structures, level of decomposition, libraries to use)

- License: GPL 2 (or 3?).

Freemind map <--> lisp (perl, python) list convertion (III)

This post presents way to push freemind map to tree list (array) structure in perl.

My first idea was to use XSLT tranformers to get result.

However, it is appears that XSLT is intended for stylesheet conversion, and W3c alarmed, that it is not for transforming one XML of general form. And my first investigations of XPath showed that it is not trivial to process branches one after one in order.

It is easily to get subtree of already defined structure with XSLT, however it seems that it is not intended for processing of tree's with undefined structure.

Thats way, tried to implement it in perl,
with usage of the XML::LibXML.

I have succeeded to install perl XML modules on Gutsy only after switching to English Locale:

export LANG=en_US.UTF-8
sudo apt-get install libxml-libxml-perl

The program in listing prints nodes as plain text:

sub main {
# do processing
my $parser = XML::LibXML->new;

my $doc    = $parser->parse_file('map.mm')
                or die "can't parse file: $@";

print "doc parameters: ref:" . ref($doc); 
## prints all map
#print $doc->serialize();

my @nodelist = $doc->getElementsByTagName("node");

foreach my $node (@nodelist) 
        {
        my @temps=();
        foreach my $attr ($node->attributes())
                {
                print ":". $attr->value()  .":"; 
                };
        print "[" . $node->getAttribute("TEXT")   . "]\n";
        }
}

main();

Output is:

:1203720731508::Freemind_Link_208208947::1203720850994::map:[map]
:1203720752265::_::1203720864185::right::check:[check]
:1203720869928::Freemind_Link_1814940840::1203720883570::check1:[check1]
:1203720883879::Freemind_Link_1671485576::1203720886018::check2:[check2]
:1203720886554::Freemind_Link_1788632798::1203720888590::check3:[check3]
:1204847625761::Freemind_Link_1338835501::1204847634916::ch3ch1:[ch3ch1]
:1204833937558::Freemind_Link_1918171523::1204833942841::left::node:[node]
:1204833944619::Freemind_Link_201749731::1204833962514::subnode1:[subnode1]
:1204833951408::Freemind_Link_749106103::1204833962515::subnode2:[subnode2]
:1204847691307::Freemind_Link_659319167::1204847697178::right::trnode:[trnode]

If you want
to process map recursively, to package or present in another tree structure, following function have to be used:

@nodes = $node->childNodes

Logic operators: Lazyness C++, Perl, Java, Javascript, ksh

Examples demonstrating lazy evaluation for || and && in different languages presented.

Consider following c++ code:

Fun1() || Fun2() || Fun3();
Fun1() && Fun2() && Fun3();

In c++, they are executed, only if the result of overall execution depends from execution of function that was not yet executed. '&&' , '||' and ',' are only c++ operators, thats order is direct from left-to right.

In examples:

Java1.6:

/**
 * @author rtg
 *
 */
public final class main {
 public static boolean fun1(){System.out.println("fun1"); return true;}
 public static boolean fun2(){System.out.println("fun2"); return false;}
 public static boolean fun3(){System.out.println("fun3"); return true;}
 /**
  * @param args
  */
 public static void main (String[] args) {
  boolean res;
  System.out.println("|| test:");
  res = fun1() || fun2() || fun3();
  System.out.printf("res:%b\n",res);
  System.out.println("&& test:");
  res = fun1() && fun2() && fun3();
  System.out.printf("res:%b",res);
 }
}

Results are the same as for

Javascript in firefox 2 is lazy too:
example

Perl's operators are lazy too:

$ cat lazy_or.pl
#!/usr/bin/perl

sub fun1 {print "fun1\n"; return 1;};
sub fun2 {print "fun2\n"; return 0;};
sub fun3 {print "fun3\n"; return 1;};

sub main
{
print "or:\n";
print "res=" . (fun1() or fun2() or fun3()) . "\n";
print "and:\n";
print "res=" . (fun1() and fun2() and fun3()) . "\n";
print "&&:\n";
print "res=" . (fun1() && fun2() && fun3()) . "\n";
print "||:\n";
print "res=" . (fun1() || fun2() || fun3()) . "\n";
}

main();

rtg@kubic-roman:~$ perl lazy_or.pl
or:
fun1
res=1
and:
fun1
fun2
res=0
&&:
fun1
fun2
res=0
||:
fun1
res=1
rtg@kubic-roman:~$ perl -v

This is perl, v5.8.8 built for i486-linux-gnu-thread-multi

ksh example at the end is different in results from perl c++ and javascript.
After ksh testing I understand, why Java not accepts int's as booleans for the operator.

ksh interprets 0 as true and 1 as false, so it is the place were you have to be patient,
when swith to:

$ cat lazy.ksh
#!/usr/bin/ksh

fun1() {
echo "fun1"
return 1;
}

fun2() {
echo "fun2"
return 0;
}

fun3() {
echo "fun3"
return 1;
}

status(){
  echo $?
}

fun1 || fun2 || fun3
status
fun1 && fun2 && fun3
status
rtg@kubic-roman:~$ ./lazy.ksh
fun1
fun2
0
fun1
1

Thats strange, that a lot of programmers in presented languages where very confident about non-laziness of the operators.
However, it is common for all presented languages, and it is no conceptual difference in these languages at the point described.

C++ example:

#include 

int fun1(){std::cout << "fun1\n"; return 1;}
int fun2(){std::cout << "fun2\n"; return 0;}
int fun3(){std::cout << "fun3\n"; return 1;}

int main()
{
 int test_val;
 std::cout << "\n || test:\n";
 test_val = fun1() || fun2() || fun3();
 std::cout << "result is:" << test_val << std::endl;
 std::cout << "\n && test:\n";
 test_val = fun1() && fun2() && fun3();
 std::cout << "result is:" << test_val << std::endl;
}

result of code execution is:

 || test:
fun1
result is:1

 && test:
fun1
fun2
result is:0

Perl sort, use<->require, Browser passed HTTP header

1) perldoc -f sort
gives overview on sort function.

@sorted = sort {$a cmp $b} @str_list; # sort strings
@sorted = sort {$a <=> $b} @num_list; # sort nums
functions passed have to return -1,0,1 as '<=>' and 'cmp' operators do.

$a and $b are global variables, as described.

effects of application of numerical to strings:

$ perl -e '
> @s=("19","12","22","21","5");
> @s=map {++$_} @s;
> print @s; print "\n";
> print sort {$b cmp $a} @s;
> '
201323226
623222013
~$ perl -e '
@s = (19,12,22,21,5);
@s=map {++$_} @s;
print @s; print "\n";
print sort {$b <=> $a} @s;
'
201323226
232220136
$perl -e '
@s=("19","12","22","21","005");
@s=map {++$_} @s;
print @s; print "\n";
print sort {$b cmp $a} @s;
'
20132322006
23222013006
$perl -e '
@s=("19","12","22","21","005");
@s=map {++$_} @s;
print @s; print "\n";
print sort {$b <=> $a} @s;
'
20132322006
23222013006

bold marked are different and interesting.

2) use and require:
Here are c++ rough equivalents for this keywords:

# require:
# perl
require One::Two;
// C++
#include <One/Two>
---------%<------------
# use:
# perl:
use One::Two 
from_two_func();
// C++ 
#include <One/Two> // all functions in Two are in the namespace Two
using namespace Two;   //

3) Looking for browser passed parts in HTTP header.
Check CGI.pm description.
this parts are in $ENV{'[KEY]'} array.
for example, for key [KEY] == REQUEST_METHOD values = POST, GET, PUT...

Freemind map <--> lisp (perl, python) list convertion (I,II)

Intro:

I am using Freemind on regular basis, for storing ideas, notes, and planning.It is an ideal tool for the manipulating tree like structures as simple, as it can be done with plain text, and even simpler, due to the Freemind intuitive shortcuts.

In fact, file in Freemind .mm can be processed as XML (if ignore some details, like header, and html that can present inside nodes, maybe something another can be also) using XSLT transform, or in any programming language with access to XML parsing.

I want lists structures being presented by Freemind, and back,
Freemind trees being presented in list-like structures, like following: ((1 2 ) 3 (4 (5 6)) 7 8).

Part I. list to freemind converter.

I have simply modified one of previously presented programs, there : nested list implementation

Instead of writing "[" I wrote <node> tag with empty TEXT attribute, simply for anonymous fork inside Freemind:

$ret .= "<node ID=\"Perl_GEN".$id."\" TEXT=\"".$listp."\" />\n";

Also beginning and header of freemind map written in corresponding places of program.
That is

<map version="0.9.0_Beta_8">

and closing of the tag.

For the list:

$al = [1, [2, [3], 4], 5];

I have obtained:

Part II. Freemind to plain list converter.

Lets start from presentation of Freemind tree nodes in one not nested lisp list (no round brackets, '(' and ')' inside list):

Lets look inside of Freemind map:
$ cat map.mm | head
First too lines describe program and format.
Than <node> tags follow.
We need only the text field from node.

So XSLT transformer of mm to plain list, ignoring tree structure can look like following:
xslt simple.

$ xsltproc map_lisp_list_s.xsl ~/map.mm | perl -e "<>; while(<>){print;}"
(
"map"
"check"
"check1"
"check2"
"check3"
"ch3ch1"
"node"
"subnode1"
"subnode2"
"trnode" )

Perl was simply used to strip one line from beginning.

Part III,

that is Freemind to tree list will be described further.

Links: differences in syntax: Perl, tcl, Shell, C++, Python, Java, Javascript, Lisp

I have already mentioned in my post on comparing numbers and strings in shell, about problems of simultaneous use of different programming languages.

To conclude, having reference cards with description one language to another differences can be useful.

Listed is set of resources, intended for migration from one language to another.

Recommended (short,self-descriptive, useful):
languages comparison:
http://merd.sourceforge.net/pixel/language-study/syntax-across-languages/

Recommended, but not short:
Wikipedia page:
http://en.wikipedia.org/wiki/Comparison_of_programming_languages
Open directory listing for comparisons:
http://www.dmoz.org/Computers/Programming/Languages/Comparison_and_Review/
PLEAC - Programming Language Examples Alike Cookbook
Comparison of productivity of writing in different programming languages:
page.mi.fu-berlin.de/~prechelt/Biblio/jccpprtTR.pdf

Useful:
Java for c++ Programmers:
http://pages.cs.wisc.edu/~hasti/cs368/JavaTutorial/
http://triton.towson.edu/~mzimand/os/Lect2-java-tutorial.html
Lisp to javascript converter, descriptive.
http://javascript.crockford.com/little.html
Another one Lisp to Javascript converter written in javascript.( You can look into source to look into the code)
http://www.joeganley.com/code/jslisp.html

Comparison Python with Java, Lisp i.t.c.

http://wiki.python.org/moin/LanguageComparisons

Three scripting concurrents:
http://mjtsai.com/blog/2002/11/25/perl_vs_python_vs_ruby/

Accumulator generator in different languages:
http://www.paulgraham.com/accgen.html

Tcl vs. Python, with nice short examples

http://homepages.cwi.nl/~sjoerd/PythonVsTcl-old.html

This thread describes differences between bash and perl.
http://www.perlmonks.org/?node_id=661859

And at the end resource with language comparison in action (memory, speed, size).
http://shootout.alioth.debian.org/

Javascript equivalent of perl attributes.

This javascript enabled page contains code for this article
It was checked under Mozilla Firefox 2.0.0.12

In one of my previous posts I have considered perl attributes.

Attributes are functions itself,that could be defined by user, and that have access to another function body, and can operate another functions, at the moment of definition of the functions with the attributes.

For example, when such definition met by Perl in the Perl script,

sub func_sub : attribute_func 

Perl executes attribute_func, passing func_sub to the function. Then attribute_func can change func_sub executable body, adding "header" and "footer" to the function, or alerting about definition met, for example.

Due to the fact, that JavaScript is also used widely for web applications, but on client side,
I have asked myself about equivalent JavaScript form.

It appeared, that for JavaScript it is even more simple equivalent exists, that can be used for understanding Perl attributes definition. Attribute can be implemented as high order function, that manipulates function body.

We can log function in and out using following attribute function defined as following closure:

function attr (fparam){
document.writeln("attr.beginning< br >");
fparam();
document.writeln("attr.end< br >");
}

Example of how it can be applied:

a = function(var_my)
{
 attr(function()
      {
      document.writeln("this is first line of original function (a) < br >");
      document.writeln("var passed:" + var_my + "< br >");
      } )
};

document.writeln("perform call to a() < br >");
a("1");

More complex example of attribute , that modifies code of function
before function will be executed:

We define attribute that prints function's code given, and that adds additional strings of code to function, and define this new function under the different name, and than executes modified function.

It is example of self-modifying code.

// prints a lot of debug info,
// define new function like given, but with changes in body
// and execute it.
function attr2 (fparam){
  document.writeln("attr.beginning < br >");
  document.writeln("func get:" + fparam +"< br >");
var tempstr = new String;
tempstr = fparam.toString();
tempstr = tempstr.replace(");","); global_var*=2;
document.writeln(\"inserted by attr2\");" );
tempstr = tempstr.replace("function ()",
"function b_modified()");
var func = tempstr;
document.writeln("Now call func made by attr2:"
+ tempstr + "< br >");
eval(func);
b_modified();
document.writeln("attr.end< br >");
}

// define b as attr2(anonimous function given)
b = function()
{
attr2(function()
{
document.writeln("this is first line of original function (b)< br >");
document.writeln("global_var="+ global_var);
} )
};

// global_var is printed and modified inside b
var global_var=2;
b();

In the begining of article link to the place were all the code cited
can be viewed in action published.

Perl nested lists vs. Python and Lisp.

It is no nested lists in Perl. However, Python and Lisp have this ability.

Description of implementing in Perl equivalent structure to nested Python and Lisp lists given.

Used:

$ python --version
Python 2.5.1
$ perl -v | head -2

This is perl, v5.8.8 built for i486-linux-gnu-thread-multi
$ clisp --version | head -1
GNU CLISP 2.41 (2006-10-13) (built 3371977993) (memory 3401903509)

Consider

'(1 (2 (3) 4) 5)

lisp list as the example of data structure we work with:

$ clisp
...
[1]> (setq alist `(1 (2 (3) 4) 5))
(1 (2 (3) 4) 5)
[2]> (elt alist 0)
1
[3]> (elt alist 2)
5
[4]> (elt alist 1)
(2 (3) 4)
[5]> 

Corresponding interactive session for Python:

>>> a = [1, [2, [3], 4], 5]
>>> print a
[1, [2, [3], 4], 5]
>>> print a[0]
1
>>> print a[1]
[2, [3], 4]
>>> print a[2]
5

Python in list processing is very like Lisp. The difference for defining list body is only using commas and square brackets in python, and you don't need put quote sign before it. (As done in Lisp to interpret it as list, not executing as function).

Perl is different in this point. Perl instead of interpreting list as given, removes all parens inside.

$ perl -e '@al = (1, (2, (3), 4), 5); \
 print "\@al[0]=@al[0] , \@al[1]=@al[1] , \@al[2]=@al[2] ,\
 \@al[3]=@al[3] , \@al[4]=@al[4] \n"; \
print @al; print "\n"; '
@al[0]=1 , @al[1]=2 , @al[2]=3 , @al[3]=4 , @al[4]=5
12345

Lisp has very convenient data model, all data in lisp are pointers.
And the solution of implementing nested lists in Perl is pointer based.

Pointer to list in perl uses square brackets ([]), like lists syntax in Python.
So our structure look is:

$al = [1, [2, [3], 4], 5];

Following procedure prints elements, emulating output in Python session and Common Lisp REPL:

sub slistp($)
{
my ($listp) = @_;
my $i;
my $ret = "";
        if (ref($listp) eq 'ARRAY')
        {
        $ret = "[";
        $ret .= slistp(@$listp[0]);
                for ($i=1;$i<(scalar @$listp);$i++)
                {
                  $ret .= ",";
                  $ret .= slistp(@$listp[$i]);
                };
        $ret .= "]";
        }
        else
        {
                $ret .= $listp;
        }
return($ret);
}

Usage for our list:

$al = [1, [2, [3], 4], 5];
print "\@al[0]=".@$al[0].", \@al[1]=@$al[1] , \@al[2]=@$al[2] \n";
print slistp($al);
print "\n";
print "\@al[0]=".slistp(@$al[0])."\n\@al[1]=".slistp(@$al[1]);
print "\n\@al[2]=".slistp(@$al[2])."\n";

result of complete code execution:

$ perl nestlist.pl
@al[0]=1, @al[1]=ARRAY(0x8152b44) , @al[2]=5
[1,[2,[3],4],5]
@al[0]=1
@al[1]=[2,[3],4]
@al[2]=5

So for Perl lisp-based tree structures, list references have to be used instead of lists.

Additional fees for dereferencing array pointers applied in Perl syntax.
It is no REPL integrated session in Perl, and examining complex structures based on lists require usage of additional modules, or writing own functions. Therefore Python, and especially Lisp are more preferable for list processing tasks.

perl one line hex viewer

Following can be helpful in the case, when you have to have a look on hex dump on some Unix host, you are not satisfied with od -x output, and you have no hex viewing or hex editing utilities like hexedit, hexdump, mc mode for editing hex, and even %!xxd in vi fails because all this tools are not installed on the host were your logs, traces, core dumps, images e.t.c. placed and have to be analyzed.

But you are lucky, you have perl 5.8.x installed on the host, so you can use the script written by me for the purpose described.And it can be easily modified for formatting hex output in way you like (for example, print original hex string from stdin with it hex presentaion, or reformat in any way you like)

Session below demonstrates the use of one-line perl script for displaying hex :

Blue is input
Red is comments
Green - output used for verifying
Not marked by color - output

----------------%<-------------
rtg@kubic-roman:~/develop/perl$ cat robo.log
stay
stay
stay
stay
stay
get command: help
look up
move north
problem: fuel 11%
move east
lookup

rtg@kubic-roman:~/develop/perl$ cat robo.log | perl -ne 
'map {print sprintf "%x ", ord($_)} split(//)' ; echo 
73 74 61 79 a 73 74 61 79 a 73 74 61 79 a 73 74 61 79 a 73 74
 61 79 a 67 65 74 20 63 6f 6d 6d 61 6e 64 3a 20 68 65 6c 70 a
 6c 6f 6f 6b 20 75 70 a 6d 6f 76 65 20 6e 6f 72 74 68 a 70 72
 6f 62 6c 65 6d 3a 20 66 75 65 6c 20 31 31 25 a 6d 6f 76 65 20
 65 61 73 74 a 6c 6f 6f 6b 75 70 a a
# some verification 
rtg@kubic-roman:~/develop/perl$ wc robo.log
12 18 98 robo.log
rtg@kubic-roman:~/develop/perl$ cat robo.log | 
perl -ne 'map {print sprintf "%x ", ord($_)} split(//)' | wc
      0      98     282
----------------->%------------

15 feb,2008, 0:54 initial version of article written.

Usage of perl attributes in functional programming in perl

started 13.02.2008|22:08

The idea of this article is to present basic usage of perl function attributes when programming in perl.

Official doc on attributes is in

perldoc perlsub
perldoc attributes

Two perldoc's above describe syntax of attributes. However, for the moment, they lack description of how they can be used. It seems from the results achieved from the test written, that they can be very useful for functional programming in perl.

Simple example below, presents "Hello World" in perl with attributes.

Perl 5.8.8 was used.

"attribute" for function, is basically something that can be added to function to modify it.

You can define following subs:

-------------%<--------------
sub _good ...

sub _hello ...
{
...
 print "Hello World from $package\:\:$name \n";
        $owner_of_attr->( @_ );
...
}

...
sub greatcode : _hello, _good
{    print "I am great code."; }

...
------------->%--------------

Attribute functions then modify code of the 'greatcode' function at the moment of the function definition.

After running the full code in perl (just store it and run in your perl without arguments) You will get:

...
Hello World from main::greatcode.
I am great code.
...

among other results.

<------>

finished for publish 14.02.2008|01:30
(this notes in italic just for my time-management procedures; blog eats a lot of time; however, without blog a lot will be lost in the mind.)