Static analysis tools are used to improve software quality proactively by development and QA teams.
Traditional static analysis technology (that early stage tools such as lint used to offer) used to do simple type checking & more over dumping a lot of false positives.
However, advanced/latest static analysis technology finds more valuable defects such as system crashes, memory leaks, uninitialized access, holding reference to freed memory, concurrency issues etc. Even the false positives concern seems to have been addressed in these tools. In addition, static analysis tools seem to provide some software metrics that can help team to proactively publish the quality of their code.
How do advanced static analysis tools detect bugs?
At a very high level, static analysis tools analyze various possible control flows & various possible contexts (variables) using an abstract machine. They also seem to hypothetically replace returning values of various functions/sub-program to evaluate the overall software behavior. They also seem to eliminate exploration of infeasible paths through advanced techniques. Analysis techniques are language-specific, but independent of hardware/OS used to run the software.
Each tool has different techniques & each one will find different defects on the same code base. So, there is not general standard technique used across all the tools.
At a very broad level, advanced static analysis tools can be classified into two categories:
- Sound tools: These tools sacrifice maximization of finding bugs to minimizing false positives to a reasonable level. So, these tools are intentionally imprecise & undecidable - this means possible false positives & false negatives.
- Unsound tools/Formal tools: These tools report all defects checked for & produces no false negatives. These are relatively slower. but, produces a large amount of false positives.
For non-mission-critical commercial applications, deploying sound tools result in diminishing returns for the investment made. So, a combination of "static analysis + other run time testing" is more fetching for non-mission-critical commercial applications.
What kind of defects do static analysis tools detect?
- Resource leaks - memory, socktets, fds
- memory corruption possibilities: buffer overflows, uninitialized access, use after free(holding reference to freed memory), invalid free (null/unallocated)
- Illegal operations: divide by zero, and illegally dereferenced pointers, double free, returning pointer to local variable, negative array index read,
- concurrency issues (double locking, missing unlocking, lock ordering issues (deadlocks), blocking calls/sleep in lock acquired state)
- optimizations such as dead code
- portability to other architectures
- missing returns, unprocessed return values, missing break in switch
- large pass by value function parameters, large stack use
- security vulnerabilities (buffer overruns, system calle based or exec based command injection,
Popular products in Static Analysis
- Coverity Prevent from Coverity (unsound tool) - C/C++/Java, commercial
- KlockWork K7/Insight (unsound tool) - C/C++/Java, commercial,
- CodeSonar from GammTech (unsound tool - C/C++, commercial
- Mathworks PolySpace (sound/formal tool) - commercial
- cppcheck - C++, free tool, light weight
- Lint - C/C++, open source, light weight
- PC-Lint - C, commercial, heard of generating a lot of false positives.
- FlexLink - closed source, lightweight tool compared to coverity/KlockWork
- Splint - C, free, based on lint with improvements in security/style checking, light weight
What stages of product development cycle should these tools be deployed?
- Run by developers on the diff before committing to version control system.
- Run by development & QA teams in nightly builds before & after internal release to QA.
Does defect detection using static analysis make QA testing and other run time tools/techniques (such as dynamic analysis) unnecessary?
For this reason, the real QA testing as well as the need for other run time tools to debug issues found in QA testing is very much necessary. So, static analysis tools is only additional leverage rather than either/or comparison wit QA testing and run time tools.
Static analysis tools vs dynamic analysis tools
Static analysis tools compared to dynamic analysis tools:
- Static analysis doesn't not need test cases or real test hardware to cull out the defects.
- Static analysis can analyze even parts of code practical test cases can't reach or code reviewers can spot.. so, makes static analysis a must-have tool for mission-critical software such as ...
- static analysis visualizes every possible control flows & detect defects where as dynamic analysis tools report only defect pertaining to the execution path of the test being performed.
- Dynamic analysis helps in functional tests as opposed to static analysis' hypothetical tests.
- Static analysis is faster.. does more tests than dynamic analysis in the same period of time particularly due to lack of need to run real tests.
Pros:
- Static analysis tools perform source code analysis without running it. So,
- no test team or test hardware is necessary.
- a lot of corner case tests which might be impossible or difficult to test would be exercised.
- Cheaper compared to depending on QA teams for all bugs. So, proactively using static analysis tool for basic bugs and depending on QA teams for advanced bugs is more cost effective.
- Static analysis can be done at any stage of product development cycle even before a usable product is available. So, the bugs caught at early stages of development offers planning buffer & avoid lots of debugging troubles later on.
Cons:
- Tests not related to functional requirements, integration testing requirements
- Shift of developer cycles from programming to dealing with false positives
TBD:
- Can we use static analysis defect to improve QA test suites?
- What is the approx time these tools takes for certain number of lines..1,400 lines of code per minute?
No comments:
Post a Comment