Question Regarding Best Practices on Upgrading from Adabas/Natural to a RDBMS System

, ,
1

First, I want to apologize for the lack of detail in this post. I’m looking at a database system in use by another party, and there are certain legal reasons why it is best for me not to be too specific about who that other party is. Let me emphasize that this is not because of any criminal activity on my part. I have no access to the database in question, and am having to infer details about its functional history from public domain information output by and written about it. This reticence on my part is only with respect to public forum posts, and I can give more details, if desired, via Private Message.

As it is, I have to put my question as a hypothetical:

Suppose I was employed to upgrade an ADABAS/Natural database that had been in continuous operation for an extremely long time (30 years) to a more modern SQL RDBMS system, and that this database had a built in reporting system accessible through the terminal client, which needed to be replicated.

How likely is it that my work process would include going through all of the menu options of the terminal interface to determine which needed to be replicated and which did not? How likely is it that I would review the Natural source code in its entirety before beginning development on the replacement system?

Thanks in advance and I apologize again for the imposition.

2

First, good luck, you will probably need it.

Now to your questions. Let’s start with the existing system. Is it well documented? If not, you are probably in for trouble. The people who designed the system 30+ years ago are likely retired (although they might be amenable to serving as consultants to the project). If the system is not well documented, and if the original designers are not available, you will indeed have to review all the code. There may be comments somewhere like “John’s shortcut” or “Mary’s fix”. John and Mary left the organization over ten years ago, and no one still there remembers why that code was written.

I am assuming that the old system is to be completely replaced (as in, Adabas and Natural will no longer exist at the shop). If so, you will clearly have to investigate the entire menu driven system since all options will either have to appear in the new system, or, not, if certain functionality is no longer required (but, of course, the options were never removed from the user interface).

Documentation is the key. If you have it, the project is feasible. If the documentation does not exist (or so poor that it may as well not exist), RUN, do not walk, to the exit. The project will take twice as long as the longest estimate, and cost four times as much.

3

Steve made an excellent response. I think the hypothetical that you posed assumes that you are replacing the converting the data elements and processing logic as they exist today into other technology, but in my opinion, this is not always the best choice. This system was built 30 years ago, you say, based on requirements that existed at that time. You would be more successful if you started from scratch building a new system based on today’s requirements as it is possible the existing system was not able to keep up with the growing pace of requirement change. There is no reason to spend so much effort mirroring the present system in alternate technologies only to end up replicating everything that led to the decision to replace the current system, and perhaps building a system that is just as difficult if not more so to enhance.

Part of these requirements for the new system is to map data attributes of the old system to the new data design so conversion can be possible, but as for functionality, printed reports can be replaced by web pages, pdf’s, emails, dashboards and other more useful and modern intelligence and analytical tools. You may even find it necessary and desirable to separate out transactional and BI processes and data stores, or if possible, retaining the transactional processes in the existing system and modernizing the reporting/BI side. These are just conceptual ideas demonstrating that you and this client should not limit yourselves from the get-go to the idea of copying these existing data and process logic as is when there is a real opportunity for innovation that will make the cost and effort worth doing.

4

Although I don’t believe that John intended to be insulting to the SAG community, I bristled at the suggestion that a port from Adabas to some RDBMS was an upgrade. And RDBMSs are not more modern, they simply are younger.

That’s the only “answer” that I can muster.

5

I want to thank Steve, Brian and Ralph for taking the time to respond to my initial post, and apologize for my lack of clarity, which seems to have given the wrong impression. I’m not under consideration for performing the “upgrade” myself. I wish it were anything near so straightforward. The ADABAS system in question has in fact been replaced already, and, insofar as I can tell, the replacement itself was done very competently.

I’m actually engaged in a bit of amateur forensic programming that has entailed analyzing the outputs of the database’s reporting system, and those of its replacement, in preparation for handing my case off to the appropriate authorities. I’m trying to establish that a change made to that system several years ago was made with malicious intent, in order to obscure evidence that a very serious crime had been committed by the person maintaining it at that time. It could be argued that the change was actually the result of poor documentation of the feature in question, and I’m seeking to eliminate that explanation as feasible.

That’s why I’m deliberately obfuscating the details of the system in question, because the person I have under suspicion is more than sufficiently tech savvy to have a few Google Alerts on the name of the system in question, and their former employer to which it belongs, and I’m mostly certain that person has no idea they are under suspicion, and fear that their learning of such suspicion would precipitate the destruction of potentially important evidence.

I appreciate how “tinfoil-hatty”, paranoid and bizarre this must sound, which is why I’m using my own name rather than a handle or alias in this forum. (I’m also mostly certain the person in question has never even heard of me.) I’m also deliberately not including most of the rest of the evidence that has me suspicious, partly for topical clarity, mostly because my last attempt at writing it up came to something on the order of 60,000 words.

I think I can explain what I’m looking at that seems so odd in the ADABAS system without making reference to specifics of persons, organizations, dates or subject matter, and will try to do so below. But we are talking about a relatively small department in a state or local bureaucracy in the United States.

I also need to apologize to Ralph in particular.

6

I do want to apologize for that slight, which was subconscious and reflects my background in Access and SQL Server development and reporting systems. I’m painfully aware that my own IT knowledge is lacking, being a self-taught amateur, so I’m not in position to cast aspersions on any particular technology’s superiority or inferiority. But, yes, that’s still the thought process of a SQL slinger who badly needs to work on his own humility and his appreciation of other technology he isn’t familiar with. (Especially for a liberal arts major who broke into programming via VBA, which is about as “low class” as programming can get.)

I’m very sorry.

In my defense, I’m falling into the pattern of thought set by the owners of the ADABAS system in question. There are public domain reports on the system for regulatory compliance purposes in which they refer to the system, verbatim, as “an antiquated system”, and their stated reasons for wishing to migrate away from ADABAS/Natural in a Mainframe environment was that “As technology has advanced”, “it has become difficult… to find programmers knowledgeable in Natural programming”. Note the implication that Natural is “less advanced”. “Upgrade” is also their exact term.

So in addition to my natural prejudices based on my coding experience, I was primed to think of the old system as “bad” based on the literature I was reading about it.

Hopefully that explains without excusing.

7

OK. Let’s see if I can explain what I’m seeing without reference to any specifics.

Note that I have all this collated and can provide examples from the reporting system’s output to anyone sufficiently interested in looking at them via private message. I just don’t want to allude to the name of system, its owners or anything that could identify it in a public post.

We’re talking about an ADABAS/Natural database belonging to a small (US) state or local government agency, that was in continuous operation from circa 30 years ago to within the last five, when it was replaced. I assume the replacement to have been RDBMS, but don’t know anything about it, and it isn’t actually relevant. This database had a custom reporting system that was accessible from the mainframe terminal client. It was developed circa 20 years ago.

There are three report types that system could output, which I’ll call 1), 2) and 3). 1) and 2) were generated on a monthly basis, usually, while 3) was run about one time per year. The agency was legally obligated to publish 3) by local statute. Each of the report types could be generated as HTML, or displayed in the terminal client and then printed to a local printer.

From circa 20 years ago to circa 15 years ago, only the latest monthly HTML copies of 1) and 2) were available on the agency’s website, while printed versions of each month’s reports were collated in three ring binders and filed for future reference. This backlog of printed versions was scanned and the scans made available through the website circa 10 years ago. Between circa 15 years ago and circa 10 years ago, 1) and 2) were run and uploaded to the website manually. Since circa 10 years ago, the reports have run automatically on 3rd of every month.

From circa 20 years ago to circa 10 years ago, report 3) was published on the website in its HTML format. Then, suddenly from circa 10 years ago to the system’s replacement within the last 5, the 3) reports became scans of print outs. The main difference between the printed version of 3) and the HTML version was that there was no legend and some of the labeling was less clear. In the new system, rather than the formatted HTML used for reports 1) and 2), which emulates the HTML formatting of the old system, 3) is now output as a custom PDF which emulates the old system’s printout version.

By dint of looking at archived copies of that agency’s staff directory on its webpage, I know who was responsible for developing and maintaining the system. I don’t know who they contracted to build the new one though. Since this post is already WAY too detailed and long, I’ll gloss over those details.

The basic question circles back to the original post though. If the ability to generate report 3) as HTML was extant in the old system’s user interface when the new one was being developed, how incompetent would the person doing that development have been not to find and incorporate it into the new system? Even if the documentation on how to use the function had been lost. If it was not extant in the UI but the feature had only been commented out in the source instead of deleted completely, would a source code review might have found it have been likely?

The upshot is that if function was not extant, it had to have been deleted, and there seems to be no legitimate reason to do that.

I know next to nothing about ADABAS/Natural or the best practices for migrating systems, but I know if I was developing a new SQL Server based systems, I would check what the old systems UI options were.

Hope that’s comparatively clear… :roll:

8

If the crux of this is whether the person responsible for delivery of the new system was incompetent or negligent because of the lack of specific features contained in the old system, it would be necessary to establish whether such features were required in the new system, whether the project manager and/or team leads misrepresented the truth about delivering such features in the new system, if the entity signed off on acceptance of the new system, and whether the current functionality of the new system is working as it was portrayed to be during system delivery, testing and implementation. The main forensics here would focus on project documentation and communications, contracts as applicable and legal requirements as applicable, and not so much on specifics on how the old system functioned vs the new unless so stipulated in such evidence.

If you are going back further in this, as in the way the old system itself was managed, then you have other leads for forensic evidence in SOX (or SOX-like) controls (privileged user review, generic account review, change management controls, etc.), ADABAS Protection logs and Command logs (if there is a legal retention period that is still binding). If functionality degraded in a suspicious manner under the old system, did this violate legal reporting requirements, or was it intended to hide something, or was it a sign of aging systems breaking without the expertise available to restore functionality? Controls and reviews typical of a Sarbanes Oxley compliance audit would be where you can potentially find leads that would point to areas requiring further investigation.

9

One thing I wanted to follow up on… perhaps it isn’t that you are believing there is total secrecy but just don’t want this thread popping up in Google searches and alerts, because a simple Google search of your name quickly found the sorts of things you have published or that others have blogged about you on. I will abide your wishes and not mention them here. but it is easy to find and not at all secret.

It is interesting stuff, this field of computer forensics. When I was cutting my teeth as a Natural programmer in the FBI, I was fortunate enough to meet SA Fred Wynekin, ISA of the San Francisco field office, who was mentioned in Clifford Stoll’s book “Cuckoo’s Egg”, which was the author’s account of how a small amount of CPU time that was used but unaccounted for in chargeback reports at a University was the thread that uncovered a massive hacking incident of national security interest against military sites as well. Agent Wynekin’s initial reaction to the author’s reporting of his initial finding was something like “Do you really expect the FBI to spend it’s massive resources tracking down a 75 cent discrepancy in CPU usage?” Of course, it ended up being more significant than that.

You probably have read that book, but if not, you will find it interesting, I am sure.

10

Oh, I’m quite aware that a search on my name turns up stuff, but most of it is dated and no longer on the web, and I avoided using names for the most part. I was actually rather counting on the name being searchable, by you at least. I’m relatively sure my utter ineptitude at drawing attention to my ideas means no one involved in the business knows my name. The Wordpress traffic to my blog at the time I did most of my writing in 2014 was pretty clear on that point.

Even still, you have a point and I do feel rather foolish.

Even the original post’s question seems rather stupid after a weekend’s reflection. I stopped myself writing before going into a description referencing (pseudonymously) the people who I knew o have been involved in the ADABAS reporting system’s job roles, and latterly remembered that since I know all of their names and, in principle, how to reach them, and I’m planning on handing this crap off to a competent investigator, it’ll be much easier for them to verify my hypothesis on the modifications to that system by interviewing those persons than it will be to involve outside experts with no firsthand knowledge.of the system.

Oh well.

Cuckoo’s Egg does sound like an interesting read, so thanks for that suggestion at least.