I find myself loving CN so much that I want to try to save some major disk space and resources by dropping a current directory-based flat-file system into CN. Ideally, this will import the existing data. I'm trying to use txtdir2xml app to build the import file, but it's providing an error:
Code: Select all
Traceback (most recent call last):
File "c:\python32\lib\site-packages\cx_Freeze\initscripts\Console3.py", line 27, in <module>
File "txtdir2xml.py", line 100, in <module>
File "txtdir2xml.py", line 17, in main
File "txtdir2xml.py", line 44, in convert
File "txtdir2xml.py", line 56, in convertFiles
File "txtdir2xml.py", line 69, in addTextFileToXml
File "C:\Python32\lib\codecs.py", line 300, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
I was prepared to split it into 256 groups and then import them one by one with:
Code: Select all
FOR /L %k IN (0,1,255) DO echo txtdir2xml -r -e utf-8 %k ips-%k.xml
The error doesn't indicate which file is experiencing problems, so it would take a lot of time to try to figure it out manually with the amount of files I'm looking at.
Oh, about that...the files I'm importing are IP data files generated from Nirsoft's WhosIP utility, using the -r (recursive) command line option. This output is then piped to a txt file on disk, as so:
Code: Select all
whosip -r 1.2.3.4 > c:\temp\1\2\3\1.2.3.4.txt
So anyway... I have roughly 70k files to import, consisting of 240mb of actual data and 8+gb of physical disk storage (thanks to the block size being significantly larger than the actual files). I know that porting this data into SQLite will save me a bunch of space and performance accessing and parsing the information, but I figure I may as well go all the way and put it in CN for the additional tagging and GUI stuff. I also haven't found any other easy way to port the data directly from files into SQLite, and CN seems as good as any option to make that happen.
If anyone can think of a SIMPLE way to make this happen, I would really appreciate it. Otherwise I'm going to have to write a script that'll do it.
Thanks again!