Exception triggered while trying to zip files in a java service

Hi all !

First of all, a little bit of context. I have a flow service in charge for taking csv files and performing some actions on them(extract some data basically). The service is triggered as soon as the file is located in a specific folder. Once I retrieved all I need from the file, it is moved to another folder and zipped. As I didn’t know how to use WmRoot.server.replicator:zip, I use a java service to zip my file (NB: I didn’t write the code but found it on my company repo and I can’t contact the author).

And here’s the issue, as long as I deliver my files one by one, it works correctly but if I try to deliver several files at the same time, I got this error for the 1st file to process:

com.wm.app.b2b.server.ServiceException: 2.null
	at ma.sap.aladin.catalog.in.priv.utils.zip(utils.java:86)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:411)
	......

I can’t figure why it happens even if I have a little intuition that it should be a thread/stream issue.

here’s the code I use to zip the file:

import com.wm.data.*;
import com.wm.util.Values;
import com.wm.app.b2b.server.Service;
import com.wm.app.b2b.server.ServiceException;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
import com.wm.data.IData;
import com.wm.data.IDataCursor;
import com.wm.data.IDataUtil;

public final class zip_SVC

{

	/** 
	 * The primary method for the Java service
	 *
	 * @param pipeline
	 *            The IData pipeline
	 * @throws ServiceException
	 */
	public static final void zip(IData pipeline) throws ServiceException {
		IDataCursor pipelineCursor = pipeline.getCursor();
		String	targetFolder = IDataUtil.getString( pipelineCursor, "targetFolder" );
		String	fileName = IDataUtil.getString( pipelineCursor, "fileName" );
		pipelineCursor.destroy();
		// pipeline
		
		if (targetFolder.charAt(targetFolder.length()-1)!='\\') targetFolder+="\\";
		
		ZipOutputStream zipOut = null;
		FileOutputStream fos = null;		
		FileInputStream fis = null;
		
		String pathPieces[]=fileName.split("\\\\");
		String nameWoPath=pathPieces[pathPieces.length-1].substring(12);
		
		String zipName=targetFolder+nameWoPath+"_zip.zip";
		
		try {
		
			fos = new FileOutputStream(zipName);
			zipOut = new ZipOutputStream(fos);
			File fileToZip = new File(fileName);
			fis = new FileInputStream(fileToZip);
			ZipEntry zipEntry = new ZipEntry(fileToZip.getName());
			zipOut.putNextEntry(zipEntry);
			byte[] bytes = new byte[1024];
			int length;
		while((length = fis.read(bytes)) >= 0) zipOut.write(bytes,0,length);
		
		} catch (Exception e) {
			e.printStackTrace();
		throw new ServiceException("1."+e.getMessage());
		} finally {
			try {
				zipOut.finish();
				zipOut.flush();
				
				fis.close();
				fos.close();	
				zipOut.close();
			} catch (Exception e) {
				e.printStackTrace();
				throw new ServiceException("2."+e.getMessage());
			}
		}
		
		// pipeline
		IDataCursor pipelineCursor_1 = pipeline.getCursor();
		IDataUtil.put( pipelineCursor_1, "zipName", zipName);
		pipelineCursor.destroy();
			
}

If someone has an idea, it would be very helpful.
Thanks !

EDIT: I finally find the solution, it seems that in some cases the java services was called before the end of the previous treatment which means it can retrieve the location of the file and cause the NullPointerException. I simply fix it with a Thread.sleep(500)just before zipping the file

Hi Gauthier,

WmRoot.server.replicator:zip ist not meant to be used directly by custom code and has a different scope than just zipping files.

You can check the Tech Community for the PSUtilities package which has some zip services included or you can rely on zip utilities provided by your OS by using pub.flow:executeOSCommand.
See IS Built-In-Services Reference for further informations on the executeOSCommand service.

Regards,
Holger

2 Likes

Hi Holger,

Thanks for your fast reply !
Unfortunately, PSUtilities is not an option (customer’s choice) but I will take a look on pub.flow:executeOSCommand.

Have a very good day :slight_smile:

Hi Gauthier,

you don´t need to use the package PSUtilities directly but you can check how the zip services were implemented to see if you can modify your code somehow to achieve what you want.

Regards,
Holger

1 Like

I would suggest avoiding executeOSCommand. The example in PSUtilities is the thing to follow.

We too have a policy of never using PSUtilities directly. For services we find useful, we copy them to our own “Public” package, edit them as desired, and use them. As @Holger_von_Thomsen notes, you can at least just reference PSUtilities to see how to do the work.

The code you posted has no mechanism for multiple files. Are you calling the Java service repeatedly, once for each file? The exception is thrown in the try of the finally block – something bad happening during finish/flush/close. Alas, no exception message available. Might need to change that block so you can what the exception is.

Did you manage to find a solution,
I have an example java service to zip files if you are interested.

You can find it on GitHub here.

The java service is here

and called

public static final void zipit (IData pipeline)
        throws ServiceException
	{
		// --- <<IS-START(zipit)>> ---
		// @sigtype java 3.5
		// [i] field:0:required sourceFolder
		// [i] field:0:required zipFile
		// [o] field:0:required zipFile
		// pipeline in
		
		IDataCursor pipelineCursor = pipeline.getCursor();
		String sourceFolder = IDataUtil.getString(pipelineCursor, "sourceFolder");
		String zipFile = IDataUtil.getString(pipelineCursor, "zipFile");
				
		// process
				
		System.out.println("build dir is " + sourceFolder + ", preparing zip file " + zipFile);
				
		try {
			if (!zipFile.startsWith("/") && !zipFile.startsWith("./"))
				zipFile = new File(new File(sourceFolder).getParent(), zipFile).getAbsolutePath();
				
			if (!zipFile.endsWith(".zip"))
				zipFile += ".zip";
				
			new Zipper().zipIt(sourceFolder, zipFile);
				
			byte[] contents;
				
			contents = Files.readAllBytes(FileSystems.getDefault().getPath(zipFile));
		} catch (IOException e) {
			e.printStackTrace();
					
			throw new ServiceException("Cannot read zipped file '" + zipFile + "' " + e.getLocalizedMessage());
		}
				
		// pipeline out
				
		//IDataUtil.put(pipelineCursor, "contents", contents);
				
		if (zipFile != null)
			IDataUtil.put(pipelineCursor, "zipFile", new File(zipFile).getName());
				
			pipelineCursor.destroy();
		// --- <<IS-END>> ---

                
	}

leverages a private class in the shared section

	private static class Zipper {
		
		private String _sourceFolder;
		
		private List <String> fileList = new ArrayList<String>();
		
		public void zipIt(String sourceFolder, String zipFile) {
			
			this._sourceFolder = sourceFolder;
		
			this.generateFileList(new File(sourceFolder));
				
		    byte[] buffer = new byte[1024];
		    String source = new File(sourceFolder).getName();
		    FileOutputStream fos = null;
		    ZipOutputStream zos = null;
		    try {
		        fos = new FileOutputStream(zipFile);
		        zos = new ZipOutputStream(fos);
		
		        FileInputStream in = null;
		
		        for (String file: this.fileList) {
		            ZipEntry ze = new ZipEntry(source + File.separator + file);
		            zos.putNextEntry(ze);
		            try {
		                in = new FileInputStream(this._sourceFolder + File.separator + file);
		                int len;
		                while ((len = in .read(buffer)) > 0) {
		                    zos.write(buffer, 0, len);
		                }
		            } finally {
		                in.close();
		            }
		        }
		
		        zos.closeEntry();
		        System.out.println("Folder successfully compressed");
		
		    } catch (IOException ex) {
		        ex.printStackTrace();
		    } finally {
		        try {
		            zos.close();
		        } catch (IOException e) {
		            e.printStackTrace();
		        }
		    }
		}
		
		public void generateFileList(File node) {
		    // add file only
		    if (node.isFile()) {
		        fileList.add(generateZipEntry(node.toString()));
		    }
		
		    if (node.isDirectory()) {
		        String[] subNote = node.list();
		        for (String filename: subNote) {
		            generateFileList(new File(node, filename));
		        }
		    }
		}
		
		private String generateZipEntry(String file) {
					
		    return file.substring(_sourceFolder.length() + 1, file.length());
		}
	}

Hi reamon,
Thanks for your answer, indeed the service is called once for each file.

For now I didn’t succeed to get more infos about the exception. What I know is it’s triggered on the 1st file to be process.

I also find that if I configure the trigger responsible for calling the service to retry after failure, the process will fail on the 1st file as expected but on the retry it works.

This sounds like a multi-threading item. Is that what is happening?

Beware timing solutions being solved by sleep(). Can have unintended consequences.

Would it work if the code you’re using was updated to support multiple filenames at one time? Would avoid multiple open, write, close cycles. And timing issues.

But if the addition of files to the archive is indeed async and files that are to be placed into the archive can “arrive” at any time, including at the same time via different threads, then you’ll want to implement something that supports that.