ModulaTor logo, 7.8KB

The ModulaTor

Oberon-2 and Modula-2 Technical Publication

Ubaye's First Independent Modula-2 & Oberon-2 Journal! Nr. 76, Jan-1999

MaX V5

Migration of 32 bit Modula-2 programs to 64 bit on OpenVMS Alpha

© (1999) by Günter Dotzel, ModulaWare

2nd edition, 16-Feb-1999

Introduction

This article is a follow-up of the implementation notes MaX V5: 64 bit Modula-2 for OpenVMS Alpha.

Remember, a 64 bit Modula-2 compiler offers you SIZE(INTEGER) = SIZE(CARDINAL) = SIZE(ADDRESS) = 8.

You might not want to migrate your applications to 64 bit, because you don't need the very large 64 bit address space. There might however be some reasons to consider a 64 bit migration:

What is a 64 bit Modula-2 compiler?

A 64 bit Modula-2 compiler is able to generate code for application programs, which want to exploit the 64 bit address space (very large memory, VLM).

One precondition to use VLM is that addresses, pointers, storage allocation, and all address arithmetic must be 64 bit wide.

From natural, pragmatic, and practical reasons, a 64 bit compiler requires that the pervasive whole number data types also have 64 bit size, otherwise existing source code, even if previously considered portable, would no longer compile.

Why? Assume for a moment that SIZE(ADDRESS)=8 and that SIZE(INTEGER)=SIZE(CARDINAL)=4 and lets examine two examples:

  1. Look at the test_VLM example below and you'll notice, that the variable "i" can not span the array index range, once "amax" gets greater than MAX(INTEGER), which it became already. Not only do you need to modify the source code, but you'd also need to import the type SIGNED_64 from module SYSTEM in a module like test_VLM, which otherwise would not need module SYSTEM.

  2. Look at the definition of the procedure WriteRecord below. The formal parameter "size" is of type CARDINAL and a 32 bit size would restrict the number of storage locations (LOC) to be transfered to MAX(CARDINAL)=4294967295, although such a restriction is not required.

What can be stored in the 64 bit space?

The underlying operating system might restrict the use of VLM. For example, the OpenVMS Linker restricts VLM to dynamic data only, which in Modula-2 terms means heap. The heap can be used to allocate large arrays. Example:


  MODULE test_VLM;
  FROM Storage IMPORT ALLOCATE;
  CONST
    amax=12345678901;
  TYPE
    Arr = ARRAY [0..amax] OF LONGREAL;
  VAR
    p: POINTER TO Arr;
    i: INTEGER;
  BEGIN
    NEW(p);
    FOR i:=0 TO amax DO
      p^[i]:=VAL(LONGREAL, i);
    END;
  END test_VLM.
The 64 bit Modula-2 compiler itself does not use VLM. In fact, MaX V5 is a 32 bit application, which runs under OpenVMS V6 or later.

The code generated by MaX V5, independent from the compilation command option /pointersize=32 or =64, can be linked for example under OpenVMS V6.1. The resulting program can be executed under OpenVMS V6.x or V7.x, under the provision that sufficient virtual memory and process quota are available.
The above example test_VLM can't be run without VLM, because the size of the array to be allocated is larger than the 32 bit address space. In fact, the array requires about 80 times the size of the 32 bit address space.

Assuming that NEW substitutes the standard Storage.ALLOCATE, any storage allocated with NEW, is allocated in the
- 32 bit address space when run on OpenVMS 6.x,
- 64 bit address space when run on OpenVMS Alpha 7.0 or later with /pointersize=64.

The OpenVMS version is detected in the body of the module Storage. If both, /pointersize=64 and VLM is present, ALLOCATE uses LIB$GET_VM_64, otherwise it uses LIB$GET_VM.

Is direct 64 bit Input/Output supported?

Yes! Being able to do direct input and output from and to VLM is important, not only for performance reasons. If direct 64 bit I/O were not possible, data in a buffer located outsize the 32 bit virtual address space must be transfered via a buffer located within the 32 but virtual address space. Also, since the OpenVMS V6.x RMS (record management services) restrict the data size per transfer to 65535 (64KB) bytes, larger transfers must split-up into several I/O-operations. OpenVMS V7.0 or later allows 2147483647 bytes (2GB) per transfer from anywhere in the 64 bit virtual address space. (See the OpenVMS Alpha Guide to 64 bit Addressing for restrictions with the SYS$PUT service routine when writing to unit record devices which do not support 64 bit addresses.)

As read- and write-operations are symetrical, we only consider writing below, using procedure WriteRecord of the Modula-2 library module FileSystem. Its definition and implementation module were compiled with /pointersize=64.

The formal parameters of the procedure FileSystem.WriteRecord are 64 bit compatible, i.e., there are no 32 bit restrictions.


  (* write an entire record to f *)
  PROCEDURE WriteRecord(
    VAR f   :File; 
    VAR rec :ARRAY OF LOC; 
      size  :CARDINAL
  );
If the storage device, holding a (previously opened or created) file f, has enough free capacity, the statement

  FileSystem.WriteRecord (f, p^, SIZE(Arr));
would write the array data referenced by p^ with size SIZE(Arr) directly from VLM to a storage- or network-device.

How does 64 bit Input/Output work?

To prove that 64 bit direct I/O works, let's go down the call hierarchy and follow the data until the control is passed to RMS. Among other statements, which do not touch the data to be transfered, procedure FileSystem.WriteRecord executes the statement


  RMSFiles.Put(f^.rab, SYSTEM.ADR(rec), size, status );
The formal parameters of RMSFiles.Put are 64 bit compatible. Below is the implementation of RMSFiles.Put. Remember, that SYSTEM.ADDRESS = SYSTEM.ADDRESS_64 with /pointersize=64.

  PROCEDURE Put (
    VAR s      :STREAM;
      record   :SYSTEM.ADDRESS;
      size     :CARDINAL;
    VAR status :CARDINAL
  );
  BEGIN
    WITH s DO
      IF os64 THEN
        RBF64 := record;
        RSZ64 := size;
        status := VMS$.SYS$PUT(Adr32(s), 0, 0 );
      ELSE
        RBF := record;
        RSZ := SYSTEM.CAST(SYSTEM.SHORTWORD,size);
        status := VMS$.SYS$PUT(SYSTEM.ADR(s), 0, 0 );
      END;
    END
  END Put;
In procedure Put, the boolean "os64" indicates that we are runing under OpenVMS Alpha 7.0 or later. In this case, we use the 64 bit record extensions fields RBF64 (64 bit record buffer address) of type ADDRESS_64 and RSZ64 (64 bit record size) of type UNSIGNED_64 of the RMS' RAB (record access block), independent from the /pointersize. RAB is a record type and a synonym of the type RMSFiles.STREAM.
In 32 bit mode under OpenVMS Alpha V6.x, with os64=FALSE, we use RBF of type ADDRESS_32 and RSZ of type SHORTWORD, which is restricted to maximum 65535 bytes.

SYS$PUT transfers control directly to the device driver of the storage device (or network, if that matters).

RAB must be allocated in the 32 bit address space. This is a restriction of OpenVMS V7.x. The caller of procedure RMSFiles.Put is responsible for the allocation of RAB. In our case this is module FileSystem, where the allocation of the RAB takes place when the file is created or opened and connected to a stream. Module FileSystem uses a 32 bit storage allocation module. To check that any client of module RMSFiles also allocated RAB in the 32 bit address space, the function procedure Adr32 is used instead of SYSTEM.ADR.
Adr32 returns the address of its formal variable parameter "s", if it is a legal 32 bit address, i.e., must be sign-extended to 64 bit (MBSE) and otherwise raises the run-time error "subrange value out-of-range", when executing the VAL function in the RETURN statement of the procedure's body.
The Adr32 check is not required, since SYS$PUT would return a status value specifying the error SS$_ARG_GTR_32_BIT (Argument greater 32 bit) as function result, if the RAB passed as the first parameter in the call of SYS$PUT were outside the 32 bit address space. But this assumes that the caller of RMSFiles.Put checks the formal parameter "status", which you never know.
OpenVMS Alpha V6.x does not allocate any virtual memory outside the 32 bit addresses space. Hence a run-time check of the address of variable "s" in case of os64=FALSE is not required.

Here is the implementation of Adr32:


  PROCEDURE Adr32(
    VAR s: ARRAY OF SYSTEM.LOC
  ): SYSTEM.ADDRESS_64;
  VAR i: SYSTEM.SIGNED_64;
  BEGIN 
    i := ADR(s);
    RETURN SYSTEM.CAST(SYSTEM.ADDRESS_64, VAL(SYSTEM.SIGNED_32, i));
  END Adr32;       
Adr32 does not copy the data to be transfered, because "s" is a variable formal parameter.

In both cases, 32 bit and 64 bit mode, the procedure RMSFiles.Put then calls the OpenVMS RMS-routine SYS$PUT, which has the following definition in the foreign definition module VMS$:


  PROCEDURE SYS$PUT (
    %IMMED  rab : SYSTEM.ADDRESS_64;
    %IMMED  err : SYSTEM.ADDRESS_64;
    %IMMED  suc : SYSTEM.ADDRESS_64
   ): CARDINAL;
The formal parameters of type ADDRESS_64 are assignment compatible with ADDRESS and ADDRESS_32, because with /pointersize=64, the 64 bit Modula-2 compiler operates with the following internal pervasive type declarations:

  ADDRESS     = CARDINAL   [0..MAX(CARDINAL)]            
  ADDRESS     = ADDRESS_64
  CARDINAL    = UNSIGNED_64
  ADDRESS_64  = UNSIGNED_64[0..MAX(UNSIGNED_64)]
  ADDRESS_32  = UNSIGNED_64[0..MAX(UNSIGNED_32)]
  UNSIGNED_32 = UNSIGNED_64[0..MAX(UNSIGNED_32)]
Subranges are assignment compatible, if their host-types (or base-types) are identical, as is the case with ADDRESS_64 and ADDRESS_32.

The Modula-2 type compatibility rules suppose an identical subrange type for ADDRESS and CARDINAL. This implies that the types ADDRESS_32 and UNSIGNED_32 have the same subrange type. A valid 32 bit address when loaded in a 64 bit register must be in canonical form, i.e., must be sign-extended from bit 31 to bit 63 (MBSE). Whole number procedure function results are returned in register 0. This precludes, that the check for a valid 32 bit address is peformed via a conversion (VAL) or an assignment of an expression with type ADDRESS_64 to a variable of type ADDRESS_32, e.g., RETURN ADR(s), with the function result type of Adr32 being ADDRESS_32.
This is the reason why procedure Adr32 above converts the value of type SIGNED_64 to type SIGNED_32, to generate the MBSE check. The shortest possible MBSE check consists of only two Alpha instructions; it is generated for the expression expression "ADR(s)" when compiling procedure Adr32 with /pointersize=32:


  SEXTL    R24,R28         ; Sign-extend longword (synonym for LDL)
  CMPEQ    R24,R28,R28     ; R28 := ORD(R24=R28)
  BEQ      R28,MBSE_error  ; If R28=0 then branch to MBSE_error,
A valid 32 address must be in the range of

 [MIN(SIGNED_32)     .. MAX(SIGNED_32)]
 or
 [0FFFFFFFF80000000H ..      7FFFFFFFH]
although OpenVMS Alpha reserves [MIN(SIGNED_32) .. -1] for the system space, which accessable only in privileged mode.

Here is the memory layout of the virtual address space on OpenVMS Alpha 7.0 and later:


   0000 0000 0000 0000H  start 32 bit user P0/P1-space, also called
                           process private space
   0000 0000 7FFF FFFFH  end   32 bit user space; total size 2 GB 

   0000 0000 8000 0000H  start 64 bit user P2-space, which includes
                           - the private page-table space,
                           - the shared page-table space, and
                           - the system S2-space
   FFFF FFFF 7FFF FFFFH  end   64 bit system S2-space; total size about
                           17,179,869 TB, but current Alpha architecture
                           implementations restrict the physical address
                           space to 4 TB

   FFFF FFFF 8000 0000H  start 32 bit S0/S1 system space
   FFFF FFFF FFFF FFFFH  end   32 bit system space; total size 2 GB
As a second example, lets examine the ISO Modula-2 procedure RawIO.Write which is defined as

  PROCEDURE Write(
    cid: ChanId;
    from: ARRAY OF SYSTEM.LOC
  );
Unfortunately, the formal parameter "from" is a value parameter (of type open array of system storage location). This means its implementation usually copies the data to the stack by default (actually, ModulaWare's Modula-2 and Oberon-2 compiler implementations do so).

In addition this limitation also restricts the transfer size to the available stack size, which is in any case smaller than the 32 bit address space. The 32 bit address space under OpenVMS contains the system space, leaving 2147483647 bytes (2GB) to the user address space, which is further shared by various program sections for code, static variables, data descriptors, constants, and literals.

Although I'd consider the stack as dynamic memory, unfortunately, in OpenVMS V7, the main stack (and the coroutine's stack) is required to be in the 32 bit address space. Anyway, coping large data blocks is inherently inefficient, if avoidable.
The stack limitation paired with the fact, that ISO Modula-2 defines "from" as a value parameter instead of a variable parameter (as, by the way, would be required in an Oberon[-2] interface), does not allow direct writing from VLM to storage devices with the higher-level ISO Modula-2 library. The same problem exists with SRawIO.Write.
Fortunately, there is a low-level ISO Modula-2 library module IOChan; it exports the procedure RawWrite:


  PROCEDURE RawWrite(
    cid         :ChanId;
    from        :SYSTEM.ADDRESS;
    locsToWrite :CARDINAL
  );
IOChan.RawWrite could be used to write a 64 bit direct output variant of RawIO.Write, as shown with the module RawOut:

DEFINITION MODULE RawOut;
IMPORT IOChan, SYSTEM;

PROCEDURE Write (cid: IOChan.ChanId; VAR from: ARRAY OF SYSTEM.LOC);
  (* Writes storage units to cid from successive components of from. *)
  (* same as RawIO.Write, except that "from" is VAR parameter *)

END RawOut.

IMPLEMENTATION MODULE RawOut; FROM IOChan IMPORT ChanId, RawWrite; FROM SYSTEM IMPORT LOC, ADR; PROCEDURE Write(cid: ChanId; VAR from: ARRAY OF LOC); BEGIN RawWrite(cid, ADR(from), HIGH(from)+1); END Write; END RawOut.
RawOut.Write is indeed very simple, portable and yet much more powerful than RawIO.Write.

64 bit Migration

One of the 64 bit migration goals was that MaX V4 as well as MaX V5 with its 32 bit and 64 bit library should be able to co-exist in the directory identified by the logical name MOD$SYSTEM. To avoid code duplication, it is required to have a single set of library-, test-, and example-modules, which compile with MaX/MVR V4 (and MaX V5 using /pointersize=32) as well as with MaX V5 using /pointersize=64.

In preparation of the MaX V5 distribution kit, some of the [iso m2] library-, test-, and example-modules of MaX/MVR V5 were modified to make these modules 64 bit upward compatible.

Most of the modifications are simple replacements of, for example, CARDINAL by UNSIGNED_32, were previously SIZE(CARDINAL)=4 was assumed.

In the following, the term "application programs" is used, when we are talking about your Modula-2 source code.

Most of the lower-level application programs contain 32 bit dependencies. Concerning the migration of your low-level applications programs from 32 to 64 bit, you have to eliminate all 32 bit dependencies.

What effort is to be expected with the migration to 64 bit? To get a first impression, study the modifications of the source code in the 64 bit enabled version of example module CountBits.mod, which is already listed in the recently revised edition of The ModulaTor, nr. 27 How to count bits?

Here is a non-exhaustive list of constructs to look at, when making your programs 64 bit upward compatible:

  1. You have to examine every occurence of SYSTEM.CAST and variant record declaration, as well as the CAST-like use of variant record fields, whether you made any assumption about the data type size.

  2. Calling a procedure having formal parameters of system storage type are a potential source of a wide range of data type size dependency errors. For value-parameters, the Modula-2 compiler accepts different sizes of formal and actual parameter. For Example:

    SIZE(SYSTEM.WORD)=4, independent from /pointersize,

    FileSystem.WriteWord(IntegerOrCardinalorAddress_Expression)

    still writes only 4 bytes, although SIZE(INTEGER)=8 with /pointersize=8, the upper 4 bytes being ignored. Using

    FileSystem.ReadWord(IntegerOrCardinalOrAddress_Variable)

    will result in a compilation error, because the variable's size must match SIZE(WORD). If you want to read only 4 bytes, independent from /pointersize, an auxiliary variable must be used. Example:

    
      VAR i: INTEGER;
      BEGIN
        FileSystem.ReadWord(f, i); 
    
    needs to be changed to
    
      VAR i: INTEGER;
         ia: SIGNED_32;
      BEGIN
        FileSystem.ReadWord(f, ia); i:=ia;
    
    The latter works independent from /pointersize.

  3. As in (2) above, any call of a procedures which has a formal value or variable parameter of type ARRAY OF SYSTEM.(BYTE, LOC, SHORTWORD, WORD, LONGWORD, QUADWORD) must be inspected, whether there are implicit data type size assumptions.

Good luck!

Note, that most of the above also applies to migrating 32 bit Oberon-2 programs to 64 bit, except that the 64 bit OpenVMS Alpha Oberon-2 Compiler offers SIZE(LONGINT) = SIZE(SYSTEM.PTR) = 8.


The ModulaTor Forum

Education and Amusement

In January 1999, this book was the number two bestseller on Amazon:
The Motley Fool's Rule Breakers, Rule Makers by David and Tom Gardner is a business and investing book about companies who got successful, because they did break rules and how to locate winning long-term businesses. This book is the latest from their investment book trilogy. The famous Gardner brothers also wrote
The Motley Fool Investment Guide and
The Motley Fool Investment Workbook.
The Unemotional Investor: Simple Systems for Beating the Market by Robert Sheard. Until 1998, he was affiliated with the Motley Fools and is now a professional money manager.


IMPRESSUM: The ModulaTor is an unrefereed journal. Technical papers are to be taken as working papers and personal rather than organizational statements. Items are printed at the discretion of the Editor based upon his judgement on the interest and relevancy to the readership. Letters, announcements, and other items of professional interest are selected on the same basis. Office of publication. The Editor of The ModulaTor is Günter Dotzel; he can be reached at mailto:[email deleted due to spam]


[ Home | Site_index | Legal | OpenVMS_compiler | Alpha_Oberon_System | ModulaTor | Bibliography | Oberon[-2]_links | Modula-2_links | General interesting book recommendations ]

Amazon.com [3KB] [Any browser]

Books Music Video Enter keywords...


Amazon.com logo

Webdesign by www.otolo.com/webworx, 22-Feb-1999