Comparing AFL scalability in virtual- and native environment

Tämän työn lähtökohtana oli tutkia automaattisien haavoittuvuusetsintä työkalujen (fuzzereiden) skaalautuvuutta natiivissa- ja virtuaalisessa suoritusympäristössä. Tutkielma suoritettiin monitapaustutkimuksena, jossa analyysi yksikkönä toimi fuzzeri American Fuzzy Lop (AFL). Monitapaustutkimuksen ta...

Full description

Bibliographic Details
Main Author: Pasanen, Erno
Other Authors: Informaatioteknologian tiedekunta, Faculty of Information Technology, Informaatioteknologia, Information Technology, Jyväskylän yliopisto, University of Jyväskylä
Format: Master's thesis
Language:eng
Published: 2020
Subjects:
Online Access: https://jyx.jyu.fi/handle/123456789/69772
_version_ 1826225750422847488
author Pasanen, Erno
author2 Informaatioteknologian tiedekunta Faculty of Information Technology Informaatioteknologia Information Technology Jyväskylän yliopisto University of Jyväskylä
author_facet Pasanen, Erno Informaatioteknologian tiedekunta Faculty of Information Technology Informaatioteknologia Information Technology Jyväskylän yliopisto University of Jyväskylä Pasanen, Erno Informaatioteknologian tiedekunta Faculty of Information Technology Informaatioteknologia Information Technology Jyväskylän yliopisto University of Jyväskylä
author_sort Pasanen, Erno
datasource_str_mv jyx
description Tämän työn lähtökohtana oli tutkia automaattisien haavoittuvuusetsintä työkalujen (fuzzereiden) skaalautuvuutta natiivissa- ja virtuaalisessa suoritusympäristössä. Tutkielma suoritettiin monitapaustutkimuksena, jossa analyysi yksikkönä toimi fuzzeri American Fuzzy Lop (AFL). Monitapaustutkimuksen tavoitteena oli millaisella konfiguraatiolla AFL toimii parhaiten, hidastaako suoritusympäristö fuzzeria ja skaalautuuko AFL rinnakkaisajossa olemassa olevan teorian mukaan? Kirjallisuuskatsauksen perusteella tunnistettiin neljä mittaria: koodin kattavuus, löydettyjen bugien määrä, suoritusten määrä sekunnissa, sekä yhteisen bugin löytämiseen kulunut aika. Monitapaustutkimus jaettiin viiteen osaan, joista jokainen osa toistettiin natiivissa ja virtuaaliympäristössä. Osien toistaminen suoritettiin skriptaamalla. Instanssien ajaminen ei sujunut täysin odotuksien mukaisesti. Pilotti tapauksen ajaminen vaati kaksi uusintakertaa, sillä tuloksien perusteella oli nähtävissä, että AFL ei ollut käynnistynyt oikein. Tuloksia kerätessä huomattiin myös, että yhden päivän koe oli uusittava virheen vuoksi. Lisäksi yhden virtuaalisen tapauksen tuloksia ei ollut kirjattu tuntemattomasta syystä, mutta tämä ei ollut este tutkimuksen tuloksien analysoimiseksi. Tutkimuksen tulokset analysoitiin Mann-Whitney U-testillä sekä Vargha-Delaney Â12 vaikutuksen suuruus testillä. Koodin kattavuutta ei voitu arvioida, sillä tulokset olivat liian homogeenisia. Löydettyjä bugeja oli yhteensä seitsemän, mutta bugeja oli löydetty hyvin harvakseltaan, jolloin vertailua ei voitu suorittaa. Suoritusnopeuden tapauksessa konfiguraatioiden keskiarvon mittaaminen muodostui ongelmalliseksi, koska keskiarvo suoritusnopeudesta konvergoitui renkien (slave) tuloksien ympärille. Suoritusnopeutta mitattiin täten laskemalla fuzzereiden yhteenlaskettu keskiarvo. Kumulatiivisen keskiarvon lisäksi kaikista tapauksista löytyi yhteinen bugi (read_utmp) jota voitiin käyttää tehokkuuden mittaamiseen. Tuloksien perusteella voidaan todeta, että käyttäessä monta isäntää fuzzeri nopeutuu, mutta lisättäessä renkejä sen kyky löytää bugeja paranee. Vastaavasti virtuaali- ja natiivi toteutus eivät tehollisesti eronneet toisistaan merkittävästi. Lopuksi voidaan todeta, että fuzzaaminen skaalautuu erittäin tehokkaasti käyttäessä kahta tai kolmea tietokoneen ydintä. Object of this study is to explore scalability of automatic vulnerability discovery tools (Fuzzers) in virtual and native execution environments. Multiple-case study was executed while the unit of analysis within was fuzzer American Fuzzy Lop (AFL). Research questions for this multiple-case study were: Does AFL scale ac-cording to known theoretical models, how is the scalability hindered through virtualization and how does the performance differ when different AFL configurations are used? From current academia four different metrics were identified: code cover-age, bug count, execution speed and time to find shared bug. Multiple-case study was done through five cases in both native and virtual environment. Execution of cases was done through scripting. Execution of cases had few problems. Pilot study had to be repeated twice because of irregularities in data showing that AFL had not started properly. During gathering the results, it was discovered that one day worth of data had to be rerun. In addition, for unknown reason, one virtual instance run is forever lost, but it does not hamper the analysis of this study. This study used Mann-Whitney U-test and Vargha-Delaney Â12 effect size measurement to assess metrics. Code coverage proved to be homogenous and was therefore discarded. Instances found a total of seven unique bugs and there-fore results were too sparse to be analyzed. Execution speed proved to be biased as the averages of instances skewed towards larger dataset of slave configured fuzzers. Therefore, cumulative values of execution speed per configuration were used as metric. Furthermore, a single shared bug was found (read_utmp) which could be used to assess performance. Study concludes that configuration of instances favors execution speed while masters are used, while bug discovery is enhanced by using slave configured instances in addition to masters. No significant performance difference was found between virtual and native environments. Finally, it can be said that fuzzers scale well in two and three core instances.
first_indexed 2020-06-08T20:00:47Z
format Pro gradu
free_online_boolean 1
fullrecord [{"key": "dc.contributor.advisor", "value": "Costin, Andrei", "language": "", "element": "contributor", "qualifier": "advisor", "schema": "dc"}, {"key": "dc.contributor.author", "value": "Pasanen, Erno", "language": "", "element": "contributor", "qualifier": "author", "schema": "dc"}, {"key": "dc.date.accessioned", "value": "2020-06-08T10:09:11Z", "language": null, "element": "date", "qualifier": "accessioned", "schema": "dc"}, {"key": "dc.date.available", "value": "2020-06-08T10:09:11Z", "language": null, "element": "date", "qualifier": "available", "schema": "dc"}, {"key": "dc.date.issued", "value": "2020", "language": "", "element": "date", "qualifier": "issued", "schema": "dc"}, {"key": "dc.identifier.uri", "value": "https://jyx.jyu.fi/handle/123456789/69772", "language": null, "element": "identifier", "qualifier": "uri", "schema": "dc"}, {"key": "dc.description.abstract", "value": "T\u00e4m\u00e4n ty\u00f6n l\u00e4ht\u00f6kohtana oli tutkia automaattisien haavoittuvuusetsint\u00e4 ty\u00f6kalujen (fuzzereiden) skaalautuvuutta natiivissa- ja virtuaalisessa suoritusymp\u00e4rist\u00f6ss\u00e4. Tutkielma suoritettiin monitapaustutkimuksena, jossa analyysi yksikk\u00f6n\u00e4 toimi fuzzeri American Fuzzy Lop (AFL). Monitapaustutkimuksen tavoitteena oli millaisella konfiguraatiolla AFL toimii parhaiten, hidastaako suoritusymp\u00e4rist\u00f6 fuzzeria ja skaalautuuko AFL rinnakkaisajossa olemassa olevan teorian mukaan?\nKirjallisuuskatsauksen perusteella tunnistettiin nelj\u00e4 mittaria: koodin kattavuus, l\u00f6ydettyjen bugien m\u00e4\u00e4r\u00e4, suoritusten m\u00e4\u00e4r\u00e4 sekunnissa, sek\u00e4 yhteisen bugin l\u00f6yt\u00e4miseen kulunut aika. Monitapaustutkimus jaettiin viiteen osaan, joista jokainen osa toistettiin natiivissa ja virtuaaliymp\u00e4rist\u00f6ss\u00e4. Osien toistaminen suoritettiin skriptaamalla.\nInstanssien ajaminen ei sujunut t\u00e4ysin odotuksien mukaisesti. Pilotti tapauksen ajaminen vaati kaksi uusintakertaa, sill\u00e4 tuloksien perusteella oli n\u00e4ht\u00e4viss\u00e4, ett\u00e4 AFL ei ollut k\u00e4ynnistynyt oikein. Tuloksia ker\u00e4tess\u00e4 huomattiin my\u00f6s, ett\u00e4 yhden p\u00e4iv\u00e4n koe oli uusittava virheen vuoksi. Lis\u00e4ksi yhden virtuaalisen tapauksen tuloksia ei ollut kirjattu tuntemattomasta syyst\u00e4, mutta t\u00e4m\u00e4 ei ollut este tutkimuksen tuloksien analysoimiseksi.\nTutkimuksen tulokset analysoitiin Mann-Whitney U-testill\u00e4 sek\u00e4 Vargha-Delaney \u00c212 vaikutuksen suuruus testill\u00e4. Koodin kattavuutta ei voitu arvioida, sill\u00e4 tulokset olivat liian homogeenisia. L\u00f6ydettyj\u00e4 bugeja oli yhteens\u00e4 seitsem\u00e4n, mutta bugeja oli l\u00f6ydetty hyvin harvakseltaan, jolloin vertailua ei voitu suorittaa. Suoritusnopeuden tapauksessa konfiguraatioiden keskiarvon mittaaminen muodostui ongelmalliseksi, koska keskiarvo suoritusnopeudesta konvergoitui renkien (slave) tuloksien ymp\u00e4rille. Suoritusnopeutta mitattiin t\u00e4ten laskemalla fuzzereiden yhteenlaskettu keskiarvo. Kumulatiivisen keskiarvon lis\u00e4ksi kaikista tapauksista l\u00f6ytyi yhteinen bugi (read_utmp) jota voitiin k\u00e4ytt\u00e4\u00e4 tehokkuuden mittaamiseen.\nTuloksien perusteella voidaan todeta, ett\u00e4 k\u00e4ytt\u00e4ess\u00e4 monta is\u00e4nt\u00e4\u00e4 fuzzeri nopeutuu, mutta lis\u00e4tt\u00e4ess\u00e4 renkej\u00e4 sen kyky l\u00f6yt\u00e4\u00e4 bugeja paranee. Vastaavasti virtuaali- ja natiivi toteutus eiv\u00e4t tehollisesti eronneet toisistaan merkitt\u00e4v\u00e4sti. Lopuksi voidaan todeta, ett\u00e4 fuzzaaminen skaalautuu eritt\u00e4in tehokkaasti k\u00e4ytt\u00e4ess\u00e4 kahta tai kolmea tietokoneen ydint\u00e4.", "language": "fi", "element": "description", "qualifier": "abstract", "schema": "dc"}, {"key": "dc.description.abstract", "value": "Object of this study is to explore scalability of automatic vulnerability discovery tools (Fuzzers) in virtual and native execution environments. Multiple-case study was executed while the unit of analysis within was fuzzer American Fuzzy Lop (AFL). Research questions for this multiple-case study were: Does AFL scale ac-cording to known theoretical models, how is the scalability hindered through virtualization and how does the performance differ when different AFL configurations are used?\nFrom current academia four different metrics were identified: code cover-age, bug count, execution speed and time to find shared bug. Multiple-case study was done through five cases in both native and virtual environment. Execution of cases was done through scripting.\nExecution of cases had few problems. Pilot study had to be repeated twice because of irregularities in data showing that AFL had not started properly. During gathering the results, it was discovered that one day worth of data had to be rerun. In addition, for unknown reason, one virtual instance run is forever lost, but it does not hamper the analysis of this study.\nThis study used Mann-Whitney U-test and Vargha-Delaney \u00c212 effect size measurement to assess metrics. Code coverage proved to be homogenous and was therefore discarded. Instances found a total of seven unique bugs and there-fore results were too sparse to be analyzed. Execution speed proved to be biased as the averages of instances skewed towards larger dataset of slave configured fuzzers. Therefore, cumulative values of execution speed per configuration were used as metric. Furthermore, a single shared bug was found (read_utmp) which could be used to assess performance.\nStudy concludes that configuration of instances favors execution speed while masters are used, while bug discovery is enhanced by using slave configured instances in addition to masters. No significant performance difference was found between virtual and native environments. Finally, it can be said that fuzzers scale well in two and three core instances.", "language": "en", "element": "description", "qualifier": "abstract", "schema": "dc"}, {"key": "dc.description.provenance", "value": "Submitted by Paivi Vuorio (paelvuor@jyu.fi) on 2020-06-08T10:09:11Z\nNo. of bitstreams: 0", "language": "en", "element": "description", "qualifier": "provenance", "schema": "dc"}, {"key": "dc.description.provenance", "value": "Made available in DSpace on 2020-06-08T10:09:11Z (GMT). No. of bitstreams: 0\n Previous issue date: 2020", "language": "en", "element": "description", "qualifier": "provenance", "schema": "dc"}, {"key": "dc.format.extent", "value": "127", "language": "", "element": "format", "qualifier": "extent", "schema": "dc"}, {"key": "dc.format.mimetype", "value": "application/pdf", "language": null, "element": "format", "qualifier": "mimetype", "schema": "dc"}, {"key": "dc.language.iso", "value": "eng", "language": null, "element": "language", "qualifier": "iso", "schema": "dc"}, {"key": "dc.rights", "value": "In Copyright", "language": "en", "element": "rights", "qualifier": null, "schema": "dc"}, {"key": "dc.subject.other", "value": "vulnerability discovery", "language": "", "element": "subject", "qualifier": "other", "schema": "dc"}, {"key": "dc.subject.other", "value": "fuzzers", "language": "", "element": "subject", "qualifier": "other", "schema": "dc"}, {"key": "dc.subject.other", "value": "multiple-case study", "language": "", "element": "subject", "qualifier": "other", "schema": "dc"}, {"key": "dc.subject.other", "value": "scalability", "language": "", "element": "subject", "qualifier": "other", "schema": "dc"}, {"key": "dc.title", "value": "Comparing AFL scalability in virtual- and native environment", "language": "", "element": "title", "qualifier": null, "schema": "dc"}, {"key": "dc.type", "value": "master thesis", "language": null, "element": "type", "qualifier": null, "schema": "dc"}, {"key": "dc.identifier.urn", "value": "URN:NBN:fi:jyu-202006084029", "language": "", "element": "identifier", "qualifier": "urn", "schema": "dc"}, {"key": "dc.type.ontasot", "value": "Pro gradu -tutkielma", "language": "fi", "element": "type", "qualifier": "ontasot", "schema": "dc"}, {"key": "dc.type.ontasot", "value": "Master\u2019s thesis", "language": "en", "element": "type", "qualifier": "ontasot", "schema": "dc"}, {"key": "dc.contributor.faculty", "value": "Informaatioteknologian tiedekunta", "language": "fi", "element": "contributor", "qualifier": "faculty", "schema": "dc"}, {"key": "dc.contributor.faculty", "value": "Faculty of Information Technology", "language": "en", "element": "contributor", "qualifier": "faculty", "schema": "dc"}, {"key": "dc.contributor.department", "value": "Informaatioteknologia", "language": "fi", "element": "contributor", "qualifier": "department", "schema": "dc"}, {"key": "dc.contributor.department", "value": "Information Technology", "language": "en", "element": "contributor", "qualifier": "department", "schema": "dc"}, {"key": "dc.contributor.organization", "value": "Jyv\u00e4skyl\u00e4n yliopisto", "language": "fi", "element": "contributor", "qualifier": "organization", "schema": "dc"}, {"key": "dc.contributor.organization", "value": "University of Jyv\u00e4skyl\u00e4", "language": "en", "element": "contributor", "qualifier": "organization", "schema": "dc"}, {"key": "dc.subject.discipline", "value": "Tietojenk\u00e4sittelytiede", "language": "fi", "element": "subject", "qualifier": "discipline", "schema": "dc"}, {"key": "dc.subject.discipline", "value": "Computer Science", "language": "en", "element": "subject", "qualifier": "discipline", "schema": "dc"}, {"key": "yvv.contractresearch.funding", "value": "0", "language": "", "element": "contractresearch", "qualifier": "funding", "schema": "yvv"}, {"key": "dc.type.coar", "value": "http://purl.org/coar/resource_type/c_bdcc", "language": null, "element": "type", "qualifier": "coar", "schema": "dc"}, {"key": "dc.rights.accesslevel", "value": "openAccess", "language": null, "element": "rights", "qualifier": "accesslevel", "schema": "dc"}, {"key": "dc.type.publication", "value": "masterThesis", "language": null, "element": "type", "qualifier": "publication", "schema": "dc"}, {"key": "dc.subject.oppiainekoodi", "value": "601", "language": "", "element": "subject", "qualifier": "oppiainekoodi", "schema": "dc"}, {"key": "dc.subject.yso", "value": "haavoittuvuus", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.subject.yso", "value": "tehokkuus", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.subject.yso", "value": "tapaustutkimus", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.subject.yso", "value": "virtuaaliymp\u00e4rist\u00f6", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.subject.yso", "value": "vulnerability", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.subject.yso", "value": "efficiency (properties)", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.subject.yso", "value": "case study", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.subject.yso", "value": "virtual environment", "language": null, "element": "subject", "qualifier": "yso", "schema": "dc"}, {"key": "dc.format.content", "value": "fulltext", "language": null, "element": "format", "qualifier": "content", "schema": "dc"}, {"key": "dc.rights.url", "value": "https://rightsstatements.org/page/InC/1.0/", "language": null, "element": "rights", "qualifier": "url", "schema": "dc"}, {"key": "dc.type.okm", "value": "G2", "language": null, "element": "type", "qualifier": "okm", "schema": "dc"}]
id jyx.123456789_69772
language eng
last_indexed 2025-02-18T10:56:06Z
main_date 2020-01-01T00:00:00Z
main_date_str 2020
online_boolean 1
online_urls_str_mv {"url":"https:\/\/jyx.jyu.fi\/bitstreams\/273c829d-397f-4a93-9241-49c80494e436\/download","text":"URN:NBN:fi:jyu-202006084029.pdf","source":"jyx","mediaType":"application\/pdf"}
publishDate 2020
record_format qdc
source_str_mv jyx
spellingShingle Pasanen, Erno Comparing AFL scalability in virtual- and native environment vulnerability discovery fuzzers multiple-case study scalability Tietojenkäsittelytiede Computer Science 601 haavoittuvuus tehokkuus tapaustutkimus virtuaaliympäristö vulnerability efficiency (properties) case study virtual environment
title Comparing AFL scalability in virtual- and native environment
title_full Comparing AFL scalability in virtual- and native environment
title_fullStr Comparing AFL scalability in virtual- and native environment Comparing AFL scalability in virtual- and native environment
title_full_unstemmed Comparing AFL scalability in virtual- and native environment Comparing AFL scalability in virtual- and native environment
title_short Comparing AFL scalability in virtual- and native environment
title_sort comparing afl scalability in virtual and native environment
title_txtP Comparing AFL scalability in virtual- and native environment
topic vulnerability discovery fuzzers multiple-case study scalability Tietojenkäsittelytiede Computer Science 601 haavoittuvuus tehokkuus tapaustutkimus virtuaaliympäristö vulnerability efficiency (properties) case study virtual environment
topic_facet 601 Computer Science Tietojenkäsittelytiede case study efficiency (properties) fuzzers haavoittuvuus multiple-case study scalability tapaustutkimus tehokkuus virtuaaliympäristö virtual environment vulnerability vulnerability discovery
url https://jyx.jyu.fi/handle/123456789/69772 http://www.urn.fi/URN:NBN:fi:jyu-202006084029
work_keys_str_mv AT pasanenerno comparingaflscalabilityinvirtualandnativeenvironment