Here, I shall be updating relevant progress made in NLP-Engine profiling.
@dehilster @meyersamnon
1 Like
For the first week and a half, I managed to compile and build the NLP-Engine from the repo. I faced some issues along the way related to icu and vcpkg. Fixed by installing autoconf-archive package.
As far as profilers are concerned, I tried to use valgrind, because I’ve worked with it previously. Using it I’m able to get the cumulative CPU instructions executed by each module, a fair idea of resources used. I’m thinking total time by each module is a better indicator, so my plan is to explore other profilers like gperftools and the one from intel to see if they are any better.
1 Like
Thanks, Niranjan, sounds like a good start.
Would be good to know what analyzer(s) you are testing on, as we may want a suite that focuses on different parts of the NLP++ runtime. Eg, analyzers that focus on KB, on dictionary, on pattern passes, on recursive grammar passes, and so on. Some small analyzers can test these.
And some large analyzers and large KBs/dictionaries can show where scaling up causes bottlenecks.
On that note, I just had a quick question about analyzers.
I was curious on what KBs, KBBs are in general. When i go through the file, I see that they are some kind of reference materials? And how do I go about using the analyzers repository under @dehilster you have for use with the engine.
I can figure out the second part eventually, but wanted to know what the repository has and how they fit in place with the entire NLP++.
They are knowledge base files. They are a hierarchical database that NLP++ uses for knowledge. It is known as the conceptual grammar. The KB files are old, non-human readable files that were used to dump and read in knowledge. The KBB files are the new human-readable files that are now used.
1 Like
Actually .dict files and .kbb files end up in the same place: the conceptual grammar (or knowledge base). These files are used to make it easy for users to create dictionaries and knowledge.
1 Like
@inkredible10 @meyersamnon How are thing going? You writing down what you are doing everyday so you can include it in your bi-weekly report?
@dehilster yeah, i’m maintaining a timesheet in which im writing down what i’m doing. I’m planning to integrate it to the repository in the coming days.
Also, I think I should be done with running the engine with the analyzers and provide what im using for each by EOD mostly, or tomorrow by max.
Super great to hear! I am just checking so that we conform and you will not have any problems with the visa requirements.
1 Like
Hi, Niranjan,
Checking in to see how you’re doing. Let me know how I can help.
Thx,
Amnon
I was trying out with the analyzers based on the tutorial videos in visualtext. Apologies for the delay. I will definitely reach out soon regarding the run and profilers.
Hi Amnon,
I was just trying out test-nlp.exe by modifying what analyzer it uses. For now, I am using parse-en-us, corporate, and basic and managed to generate a profiler output for it using valgrind to get a feel for using nlp-engine.
I’m now thinking of using different analyzers with different sequences, and move to the bigger analyers that use more resources (with different profiler like gperftools). Do you think we are good, running our profiler on the same TAI test script or do we incorporate more diverse text?
Also, is it usual that the output of parse-en-us in the output dir is empty? Although, that is empty, i’m able to see the output tree of the pass to corporate and basic
I was also able to call the nlp-engine without test-nlp for different analyzers. This is great, now that I understand how I get this pipeline working. I see that intel’s profiler is more suited for the performance benchmarking so I will set up to get the profiler output. Will update here when I get the data.
@meyersamnon @dehilster
Good afternoon!
I was able to set up Intel vtune Profiler to analyze the bottlenecks by running it on the test-nlp. I have attached the image of one such analysis. For hotspots and checking which module consumes more time, i’d need a usecase where the executable runs for more time…thinking of calling analyzers that can consume more resources.
I did try it on one of my projects to see how the hotspot analysis would look like
1 Like
@dehilster
Hi, and thanks, looks good.
You may want to look at the parse-en-us analyzer that comes with the VSCode NLP++ extension. You can load bigger input files into the analyzer’s input directory and test that kind of run. The analyzer is pretty big, with many passes, and will run slowly with a large input file.
Let us know how that goes.
Amnon
@meyersamnon @dehilster I loaded a 6MB text file for analysis with parse-en-us and obtained the following data of memory consumption and time by each function
(result_hotspot - Google Sheets) - for CPU time analysis
(result_memory - Google Sheets) - for memory analysis
Please let me know if you have anything else in mind
I also have a question regarding parse-en-us. When I use other analysers I get some final tree generated but for parse-en-us, I don’t seem to have the tree although it runs. Do you know why is that the case?
@dehilster
Hi,
(DAVID – any thoughts for moving forward here?)
As for final parse tree, if you run with /verbose flag set (ask David how to set command arguments, don’t know how that happens with the NLP-ENGINE), you should see a parse tree log file for most passes. This can pinpoint where parse trees stop getting output – can help pinpoint a failure of the analyzer. Seems like Parse::finalTree() is getting executed, maybe you can trace in there.
As for the overall, one thing seems to be that loading the analyzer passes and rules and “interning” them seems to take a lot of time (5 seconds or so here). If we test on a large knowledge base, would be good to see if that is a bottleneck as well.
Thx,
Amnon
The test file is way too big. Make the test file fot eh parse-en-us to be one paragraph. Then slowly increase it. There will be a time when it is too big.
@meyersamnon @dehilster will consider these points, run the test and let you know. That is, tracing to where the issue of final tree non-generation for parse-en-us is and to incrementally increase the test input size.
@meyersamnon could you please clarify on how can I use these large knowledge base? Like, are they dependent on the analyzers I use and should look for analyzers that have huge KBs?
I apologize for this huge delay as i was caught up with so much and couldn’t deliver this data on time.
-
I was able to look into why the parse-en-us was not generating the final-tree, then fixed that to get the trees for each output. The Parse::finalTree that we saw in the logs previously was only for the other analyzers.
-
I was able to fix tried to step up the text input size as @dehilster suggested for the following sizes (100, 1000, 10000, 50000, 100000 lines) and have attached the links to the profiler data below.
( result_memory_100k - Google Sheets ) - for memory allocation data
( result_hotspot_100k - Google Sheets ) - for CPU time data -
For the memory analysis, I was not able to perform for file size 10000 and more as my Ubuntu diskspace runs out (it generates sqlite file of large sizes that could not be accomodated). I did not want to delay further so I have sent whatever I have done so far, and also disabled prints to console so as to speed up the profiling process.
-
I haven’t yet tried out the large KBs suggestion.