Just a quick question: is the RO SDK wire protocol protected with a CRC, or does the protocol solely rely on TCP’s error correction?
I am using binary channel + compression + AES envelope. The data flows through a Hamachi VPN and I’m not sure how reliable the TCP protocol is through such a tunnel.
Kind regards,
Arthur Hoornweg
it just protected by TCP’s error correction.
if you are using [ZLIB] compression of Binary message, it also contains some checksums and won’t be properly decoded if data was broken during transmission.
TCP should be reliable by definition, if it’s. or, the you got far worse problems. there really should not be need to add a second level of error checking. (at what point do you stop?)
I am having issues where occasionally data are received as an array full of zeroes instead of valid values.
The original file on disk where the data came from does not contain these zeroes so I’m trying to figure out if it’s either a bug in my code, a transmission error or some kind of transient error (if there was a brief period during which the record in the file actually did contain zeroes). To make matters more complicated, the data flows through a Hamachi VPN tunnel and I have no idea how reliable TCP is through such a tunnel.
Would Remobjects throw an exception if unpacking an AES envelope failed?
I would hook some of the lower-level “before-send/after-receive” events on the server and the client and log the data for comparison, to see if the zeros are there…
I’d expect your TCP stack to fail on bad data, no matte what channel it came thru. TCP sits on top of whatever the network is (be it wifi, ethernet, a VPN tunnel, or in deed smoke signals. the TCP stack should enforce data integrity at its level, no matter how the data was transmitted, and unless the TCP implementation is broken, the corruption “can’t” really happen at a lower level w/o detection. AFAICT.
it should, yes.
no, it wouldn’t.
if you specify incorrect password, it will decompress data incorrectly. it can be considered as an additional protection because we don’t store any checksum for it.
I can’t protocol+compare the data on the server and the client because I’m literally downloading a gigabyte per day in small chunks. On the server side, there’s a data acquisition system running that writes data into a multitude of sequential files (opened “shared” on the file server). I’ve written a RO SDK client/server system for streaming those newly appended bytes to a different location. But roughly about once every 100000 queries, the client software receives zeroes instead of valid data. It is rare, but very noticeable.
I’m beginning to suspect that there’s a race condition in the Windows file system. When a file grows, Windows must regularly allocate a new empty cluster at the end of the growing file. My suspicion is that such a new cluster is initialized with zeroes before actually writing the data into it, and that there’s a very brief time window during which a simultaneously reading software could “see” those zeroes.
The files aren’t supposed to have any empty blocks…
if I understand you correctly, you are combining a big file from small chunks.
this technology is used by torrent softwares for writing small chunks into one big file.
and it doesn’t require a lot of time for allocating a big file
I’m replicating sequential data files containing sensor readings. The original data file resides at an oil well and is growing at a rate of 1 KB/s. It has to be duplicated in near-realtime to the customer’s office.
A RO SDK client software in the customer’s office polls if the original data file has grown; whatever bytes were appended since the last check are downloaded. The bytes are then appended to a copy of the file in the customer’s office. This copy has to be a valid data file at all times and may not contain empty records. So the file can’t be pre-allocated and it may not be a sparse file either. It just grows sequentially.
one option would be to write your own envelope that adds a checksum, even if just for debugging/narrowing the problem down. another option, if the file can’t have zeros, is to just add a check on the server to validate that data before sending - if it has zeros, log a warning. that way when you get bad data client side, you can go check if the server had it broken too. depending on th eputocme, you’ll know where to look for your problem further…