Use Progress and ODBC on CentOS with Japanese

This is related to my previous blog about using Chinese characters with Progress, .Net and Mono: Allow Chinese characters on a Client-Server application

Now the next challenge was to get Japanese characters to work.

I first converted the Progress database to shift-jis.
I set the environment variable: SQL_CLIENT_CHARSET=SHIFT-JIS

I then experimented with several locales, e.g. ja_JP.UTF8 and ja_JP (which is an alias for ja_JP.eucjp).

I used Wikipedia to get some sample words in Japanese characters, see http://en.wikipedia.org/wiki/Japanese_writing_system.

I soon realised that with some words I would get an ODBC error:
eg. with Kanji script 金魚 (Goldfish)
and with Hiragina script, きんぎょ
If I left out the last letter, it worked.

The ODBC error was:

Saving of data failed!
Reasons:
* [[ODBC]] [unixODBC][DataDirect-Technologies][ODBC 20101
driver][20101]Internal error -4 (unknown nls services error) in SQL from
subsystem NLS SERVICES function nlsCountChars called from nc_get_strlen
on . for . Save log for Progress technical support.

A helpful person from Progress Support gave me some suggestions, and I got it to work with Progress 9.1D in the end!
I had to use another locale, ja_JP.SJIS, which did not exist by default on my CentOS 4.6 (locale -a | grep JP).

According to http://kbase.redhat.com/faq/FAQ_80_1256.shtm
you can create the locale ja_JP.SJIS with this command:
localedef -f SHIFT_JIS -i ja_JP ja_JP.SJIS

This resolved the internal ODBC error.

Another issue was that I was not aware how to properly set the font in the ini file. Finally I found a hint in the Progress manuals, to check the prolang directory for the file progress.ini which is available for each language on the client side; this has a sample font. For the Japanese, this is a font with Japanese characters. I just copied it, and it worked.

Install Chinese or Japanese Windows XP with english XP screenshots

I sometimes have the problem that I need to install an asian Windows XP. I could always install a second XP in english in parallel in order for me to understand the screens in a language that is unknown to me.

Here are some screenshots that have helped me, hopefully they will be helpful for other people too.

I have noticed that there are more screens, e.g. Japanese XP has an extra screen for the keyboard layout. Select S for the menu, and then select "101 English keyboard layout". Otherwise you will have problems later when entering Name and Organisation. It also seems you should click on the little red dot in the keyboard selection thing at the right bottom corner when entering Name and Organisation, otherwise Tabulator key does not work, and I was not able to continue from that screen


One bit of advice: the keyboard shortcuts are usually the same, so watch out for capital Latin letters. This should help you to recognise which screen you are on and find the meaning on the english screenshots.

You can find a similar gallery here: http://techrepublic.com.com/2346-10878_11-5181.html

Use the CSParser to process C# files

I wanted to generate code but at the same time parse the target file first to see if any changes have been manually made or with the graphical designer.


I found the following parser for c# code: http://www.codeplex.com/csparser; it seems very active at the moment, and it parsed my test cs file without any problems.



I had some issues first understanding how to look for the current line number and how to know the name of the classes and methods etc. But a look at the sample file cmc.cs helped a lot.

using System;
using System.CodeDom.Compiler;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Text;
using DDW.Collections;
using DDW;
using Microsoft.CSharp;

namespace test
{
  class Program
  {
    private static void PrintErrors(IEnumerable errors)
    {
        foreach (Parser.Error error in errors)
        {
            if(error.Token.ID == TokenID.Eof && error.Line == -1)
            {
                Console.WriteLine(error.Message + "\nFile: " + error.FileName + "\n");
            }
            else
            {
                Console.WriteLine(error.Message + " in token " + error.Token.ID
                    + "\nline: " + error.Line + ", column: " + error.Column
                    + "\nin file: " + error.FileName + "\n");
            }
        }
    }
    
    private static CompilationUnitNode ParseUnit(string fileName, List errors)
    {
        Console.Write("\nParsing " + fileName);
        FileStream fs = new FileStream(fileName, FileMode.Open, FileAccess.Read);
        StreamReader sr = new StreamReader(fs, true);
        Lexer l = new Lexer(sr);
        TokenCollection toks = l.Lex();

        Parser p = null;
        CompilationUnitNode cu = null;

        p = new Parser(fileName);
        cu = p.Parse(toks, l.StringLiterals);

        if(p.Errors.Count != 0)
        {
            Console.WriteLine();
            PrintErrors(p.Errors);
            errors.AddRange(p.Errors);
            return null;
        }
        return cu;
    }
    
    public static void Main(string[] args)
    {
      string filename = "c:\\myfile.cs";
      List errors = new List();
      CompilationUnitNode cu = ParseUnit(filename, errors);

      StringBuilder sb = new StringBuilder();
      cu.ToSource(sb);
      Console.WriteLine(sb.ToString());
      foreach (NamespaceNode nnode in cu.Namespaces)
      {
        foreach (ClassNode cnode in nnode.Classes)
        {
          Console.WriteLine("class name: " + cnode.Name.Identifier.ToString());
          foreach (FieldNode fnode in cnode.Fields)
          {
            foreach (QualifiedIdentifierExpression qexpr in fnode.Names)
            {
              sb = new StringBuilder();
              qexpr.Expressions.ToSource(sb);
              Console.WriteLine("Field: " + sb.ToString());
            }
          }
          foreach (MethodNode mnode in cnode.Methods)
          {
            sb = new StringBuilder();
            mnode.Names.ToSource(sb);
            if (sb.ToString() == "InitializeComponent")
            {
              Console.WriteLine("Line number of InitializeComponent starting: " + 
                                mnode.RelatedToken.Line.ToString());
              int firstLine = -1;
              int lastLine = -1;
              foreach (StatementNode snode in mnode.StatementBlock.Statements)
              {
                if (firstLine == -1)
                {
                  firstLine = snode.RelatedToken.Line;
                }
                lastLine = snode.RelatedToken.Line;
                sb = new StringBuilder();
                snode.ToSource(sb);
                string line = sb.ToString();
                Console.Write(snode.RelatedToken.Line.ToString() + ": " + line);
                
              }
            }
          }
          
        }
      }
      
      Console.Write("Press any key to continue . . . ");
      Console.ReadKey(true);
    }
  }
}

Manage webspace with Hostsharing

I am a member of Hostsharing eG, which is a German company owned by its members with the aim to take the idea of Open Source to Webhosting. You don't need to worry about backups or availability, because the webmasters do it all for you. But you still have SSH access and many other powerful tools to manage your applications that are running on the server.

Recently I had to fight off an intruder who had used a security leak in one of my php applications. He had added 700 MB somewhere, the quota command showed it, but all calls of du did not show the files. Finally I realised I have to run find / -group pac00 which shows all files that belong to my own user or my subusers, who all belong to the group e.g. pac00.

Another issue that I often have is that some files are created with PHP and they belong to user httpd. That means they don't show up at all on the quota command, which is not fair for the other users, because everyone has to pay for the used webspace.
The first step is to actually find those files. I have written the script pac-du-httpd, which goes through all the users' directories and prints the filenames that are owned by httpd to a file, and also prints the overall sum size of those files. You can find the file here: pac-du-httpd.sh

The next step is to change the ownership of those files. On the support mailing list (yes, Hostsharing eG has as much open communication as any other Open Source project) there is a very useful explanation of how to delete such files:
Support Anleitung
The basic idea is: first run the script as normal PHP with user httpd, and change the file permissions so that everybody can change them, and then to run the script again, to delete the file.
In my situation though I don't want to delete the file, but would like to change the permission. I finally got this solution to work:

<?php
# first run as php file with httpd, to give permissions (chmod 0666) then run with cgi to recopy the file

$dh = opendir('./');

while( ($file = (readdir($dh))) !== FALSE )
{
  if ($_SERVER["ORIG_SCRIPT_NAME"] == '/cgi-bin/phpstub')
  {
    copy ($file, $file.".tmp");
    unlink ($file);
    copy ($file.".tmp", $file);
    unlink ($file.".tmp");
  }
  else
  {
    chmod($file, 0666);
  }
}
?>

I first run the script to change the file permissions, and then I run it again via CGI so that it runs for the user that the files should belong to. I make a copy of the file, delete the original, and rename the copy back. This works perfectly. And for the future, I will let the PHP always run via CGI (see also Hostsharing documentation on PHP via CGI)

Another useful tool is pac-du-quota, which is referenced from the Hostsharing Documentation on Webspace.
I have extended it to also list the size of the MySQL databases, which is in my case also quite a bit of data...
Insert this in the function collectSpace just before "#Wenn User exisiteren aufsammeln":

# Mysql Datenbanken
 for DB in /var/lib/mysql/${PAC}_*; do
   let COUNTER=COUNTER+1
   TEMP=`du -s "$DB"`
   SPACE[$COUNTER]=`echo $TEMP | sed -e 's/\([^ ]*\).*/\1/g'`
   NAME[$COUNTER]="Mysql_`basename $DB`"
 done

Allow Chinese characters on a Client-Server application

I recently had to provide a working solution for Traditional Chinese for Hongkong.

It involved both ODBC on the Linux server to read correct data from a Progress database, and process that data in .net on Mono, but also on the client side to be able to read data through ODBC.

Here is the solution and also some warnings about what can go wrong:

For Mono, you need to have installed the RPM package mono-locale-extras, which installs the dll I18N.CJK.dll (CJK stands for Chinese Japanese Korean).

Make sure the Progress database is in codepage big-5.
proutil mydb -C convchar convert undefined -G 15
proutil mydb -C convchar convert big-5 -G 15
_progres -pf batch.pf -1 -b -rx -p upgrade.p -param $DLC/prolang/tch/big-5/_tran.df | cat
proutil mydb -C idxbuild all

Make sure that your pf files (or even the Progress startup.pf) contain these lines:
-cpinternal big-5
-cpstream big-5
-cprcodein undefined

For the Progress server, you need to set this variable before the server is started (note: if you have both sql and 4gl server, the first one needs all settings already):
export SQL_CLIENT_CHARSET=BIG-5

Mono would rather expect BIG5 (without the dash), so we need to set another variable so that the default encoding is correct.
Before you start the server in the Mono environment, make sure that you export this variable:
export LANG=zh_TW.big5

The following command shows what languages are available on your Linux:
locale -a
Unfortunately there is no big5 for Hongkong, and zh_HK.big5hkscs does not seem to work for me. zh_TW.big5 works fine.

This code converts the data from the codepage coming through ODBC from the Progress database into Unicode:

FDBEncoding := System.Text.Encoding.Default;
try
sqlClientCodePage := Convert.ToInt32(System.Environment.GetEnvironmentVariable('SQL_CLIENT_CHARSET'));
FDBEncoding := System.Text.Encoding.GetEncoding(sqlClientCodePage);
except on exp: Exception do
end;

function ConvertToUnicode(s: System.String): System.String;
var
bytes: array of byte;
begin
bytes := Encoding.Default.GetBytes(s);
result := FDBEncoding.GetChars(bytes);
end;

function ConvertFromUnicode(s: System.String): System.String;
var
bytes: array of byte;
begin
bytes := FDBEncoding.GetBytes(s);
result := Encoding.Default.GetChars(bytes);
end;

For the client to be able to access through ODBC, we also need to set the environment variable in Windows XP SQL_CLIENT_CHARSET=BIG-5
(Start/Settings/Control Panel/System/Advanced/Environment Variables/System or User Variables)

What kind of relationship can we have with God

Some christians seem to talk a lot about their faith as a "Face to Face" relationship with God.

I have a problem with those words, because it sounds to me much like a relationship of two equal people who have a good understanding of each other.

The bible mentions the phrase in these places (God and human being face to face):

• 1Mo (Gen) 32,30 It is because I saw God face to face, and yet my life was spared.
  • Jacob has fought with God before meeting Esau
• 2Mo (Ex) 33,11 The LORD would speak to Moses face to face, as a man speaks with his friend.
• 5Mo (Deut) 34,10 Since then, no prophet has risen in Israel like Moses, whom the LORD knew face to face,
  • This was quite a unique relationship in the bible
• 5Mo (Deut) 5,4+5 The LORD spoke to you face to face out of the fire on the mountain. (At that time I stood between the LORD and you to declare to you the word of the LORD, because you were afraid of the fire and did not go up the mountain.)
  • The whole people of Israel met God, but it could not be continued, so Moses went alone
• Job 38, 3+4 Brace yourself like a man; I will question you, and you shall answer me. Where were you when I laid the earth's foundation? Tell me, if you understand.
  • Job cannot answer God
• 1Cor 13,12 Now we see but a poor reflection as in a mirror; then we shall see face to face. Now I know in part; then I shall know fully, even as I am fully known.
  • This is a promise for the future

Now things look a bit different when we consider that Jesus became a man and so represented God to us in a way we can grasp a little.

• John 14, 9: Anyone who has seen me has seen the Father.

But again, he does not have long discussions, but invites the disciples: Come and see! Follow me! I think there is the danger of just wanting to experience God but not being willing to follow and obey him.

• John 1, 39: Come, he replied, and you will see. So they went and saw where he was staying, and spent that day with him.
• Matthew 4, 19: Come, follow me, Jesus said, and I will make you fishers of men.

We don't need to be afraid of Jesus: He is compassionate and approachable

• e.g. Luke 8, 47: Then the woman, seeing that she could not go unnoticed, came trembling and fell at his feet. [...] Daughter, your faith has healed you. Go in peace.

Jesus enjoyed blessing the children; perhaps that is how I would also describe my relationship with Jesus? I can sit on His lap, I can tell him about my little and big worries, and he will understand. When he starts talking about the things that worry him or he is busy with, I will sit there with big eyes and won't understand much, perhaps with growing maturity get more and more a better idea what is really important.

• Luke 18, 16: But Jesus called the children to him and said, Let the little children come to me, and do not hinder them, for the kingdom of God belongs to such as these.

We can enter God's presence, and need to be intimate with him. But he will still surpass our understanding

• Phil 6,7: Do not be anxious about anything, but in everything, by prayer and petition, with thanksgiving, present your requests to God. And the peace of God, which transcends all understanding, will guard your hearts and your minds in Christ Jesus.

My (personal) conclusion:

• When I follow Jesus, I will see the back of Him. My eyes need to be focused on Jesus' back. I don't want to stand still and face Him, because then I am going in the wrong direction.
• I know I need times when I sit on His lap and listen to Him and tell Him my joys and worries.
• I am looking forward to the day when I will see God face to face, and I will understand it all and know Him completely.

My first little ASP.net project using the Mono XSP server

I wanted to provide an online version for the project Delphi2C#, so that people can just paste their Pascal code on the website, and see how good the converter is for them when it displays the C# version of their code.

On Debian, you need to install:

apt-get install mono-xsp asp.net-examples

This will give you a file

/etc/xsp/debian.webapp

It already contains the configuration for the samples, that live in

/usr/share/asp.net-demos

To start the xsp server, you just run

/etc/init.d/mono-xsp start

And you will be able to access the samples at

http://YOURSERVER:8081/samples

For example to add your own aspx file, copy it to

/usr/share/asp.net-demos/DelphiNet2CSharp.aspx

The source code of that file you can see here:
Browse CVS of Delphi2C#

If you have a dll as well, copy that to

/usr/share/asp.net-demos/bin/Delphi2CSharp.dll

In your aspx file, you need to write at the top to use that dll:

<%@ Import Namespace="Delphi2CSharp" %>

To debug your application (e.g. I had compiled the DLL for .Net 2.0, but the xsp only supported 1.1 and crashed without any message to the browser):

cd /usr/share/asp.net-demos
MONO_OPTIONS=--debug && xsp --verbose

Then in the webbrowser go to

http://YOURSERVER:8080/DelphiNet2CSharp.aspx 

You will see all the Console.WriteLine output, and also any exceptions and Mono crash messages.