In response to the emphatic demand by the Secessionists led by Carl Moll and Otto Wagner for a public gallery of contemporary art as a counterbalance to the imperial collections, in 1901 Emperor Franz Joseph I agreed to the establishment of the Moderne Galerie. The opening of the state museum in May 1903 in the Lower Belvedere laid the foundations for the current Österreichische Galerie Belvedere collections. The gallery holdings, which were meant to be housed only temporarily in the Lower Belvedere, were supplemented by the works acquired since 1851 by the Imperial Royal Ministry of Culture and Education and the Vereinigung bildender Künstler Österreichs – Secession founded in 1897, including Vincent van Gogh's Plain of Auvers and Giovanni Segantini's The Bad Mothers. Private patrons such as Prince Johann von und zu Liechtenstein and the architect Alexander Hummel supported the gallery with generous donations. The gallery committee appointed initially to manage the gallery and directly answerable to the Ministry of Culture sought constantly to enlarge the collection. In 1908, for example, it acquired Gustav Klimt's The Kiss directly from the artist. In 1911 the Moderne Galerie, directed since 1909 by Friedrich Dörnhöffer, was renamed k. k. Österreichische Staatsgalerie. The collection was systematically enlarged, with acquisitions from the Middle Ages to the present, with a view to creating a representative cross-section of art in the multinational Habsburg empire. Founder members of the Staatsgalerieverein established the same year included Ferdinand Bloch-Bauer, Nikolaus Dumba, Count Karl Lanckoroński and Victor Zuckerkandl. Through its fine arts purchases, the association sought to "cultivate a feeling for indigenous art". The museum complex, renamed Österreichische Galerie in 1921, was directed from 1915 until his removal from office in April 1938 by the art historian and Schiele patron Franz Martin Haberditzl. In spite of extremely limited financial resources, thanks to his personal network of art patrons, collectors and artists, including Adele and Ferdinand Bloch-Bauer, Haberditzl managed to acquire works by major Austrian and international modernists through gifts and loans. After the First World War, when the imperial art collections were nationalized, the Österreichische Galerie acquired the premises in the Upper Belvedere as well and was restructured in accordance with the museum reform concept devised by the art historian Hans Tietze. Artworks from the Austrian Baroque and nineteenth century were transferred from the Kunsthistorisches Museum in exchange for medieval items and the transfer of prints to the Albertina. The Baroque museum was opened in 1923 in the Lower Belvedere as the first phase of this restructuring. The Galerie des 19. Jahrhunderts (gallery of nineteenth century art) was opened in the Upper Belvedere with works by Austrian and international artists and an area for special exhibitions on the ground floor. Thanks to the financial support of the Verein der Museumsfreunde, Prince Eugene's former Orangery in the Lower Belvedere was adapted for the reopening of the Moderne Galerie as a collection of twentieth-century art. The adjoining garden contained an exhibition of modern sculptures.

After the annexation of Austria to the Nazi German Reich in March 1938, director Haberditzl was removed from office and replaced by his long-standing deputy Bruno Grimschitz. His research assistant Heinrich Schwarz was forced to leave Austria on account of racial persecution. Grimschitz took advantage of the changed power structure to embark on an offensive acquisition policy, which was also connected directly to the expropriation of "Jewish" art collections by the Nazi regime. Apart from various objects from secured and seized assets, non-expropriated objects were also assigned to the public collections by order of the Reichsstatthalterei. Many purchases of works by contemporary artists demonstrate the intentions of the Nazi cultural policy to promote "intrinsically German" art. In line with the regime's philosophy, Grimschitz had the Moderne Galerie closed on 20 March 1938, and the "degenerate" art in it remained untouched. From October 1939, the Zentralstelle für Denkmalschutz (Central Monument Protection Authority) was able to use the former Orangery as a depot for secured and seized art objects. Even before the start of the first salvage transports in late summer 1939, extensive air raid protection measures were taken to prevent damage to the Belvedere buildings. As a result, the Baroque museum in the Lower Belvedere was closed from July 1939 to January 1940 and from April 1941 onwards. In a series of transports from September 1939, the main works from the Galerie des 19. Jahrhunderts and Baroque museum were taken to Kartause Gaming, Lower Danube. Because of the increasing threat of Allied air raids and the need to intensify salvage operations, hundreds of art objects were transported in 1943/44 to various depots, such as the palaces of Immendorf, Kirchstetten and Weinern in Lower Danube and the Lauffen salt mine near Bad Ischl. After the permanent closure of the Österreichische Galerie because of the "total war effort" in August 1944, the west wing of the Upper Belvedere and the grounds were severely damaged by air raids in November 1944. Further bombing of the Upper and Lower Belvedere in February 1945 left the palace and grounds in ruins. Thanks to the "complete salvaging" of the gallery contents, there were no appreciable losses. Shortly before the end of the war, however, retreating SS members destroyed the Klimt paintings from the Lederer collection stored in Schloss Immendorf and Klimt's three Faculty pictures owned by the Österreichische Galerie.

The destruction made it necessary to completely rebuild and renovate the Belvedere palaces. After Grimschitz had been removed from office in October 1945 on account of his former NSDAP membership and the interim directorship by Fritz Novotny, Karl Garzarolli-Thurnlackh was appointed director of the Österreichische Galerie in 1947. On the basis of the Austrian restitution laws, twelve artworks were restituted after 1945 to Viktor Ephrussi, Irma Götzl, Alma Mahler-Werfel, Robert Pollak, Gustav Senders, Leopold Weinstein, Paul G. Wexberg and to the French military government. The museum was officially opened to the public in 1953 after the premises had been rebuilt and renovated. As a museum of purely Austrian art, the Österreichische Galerie was a symbol of Austria's regained sovereignty. As such, the collection of nineteenth- and twentieth-century French and German art was transferred to the Kunsthistorisches Museum. In return it obtained the collection of medieval Austrian art for the newly established Museum mittelalterlicher Kunst (museum of medieval art) in the Orangery. The Baroque museum opened in the Lower Belvedere in 1953 and the Galerie des 19. und 20. Jahrhunderts (gallery of nineteenth- and twentieth-century art) in the Upper Belvedere a year later. In keeping with the original intention of the Moderne Galerie in 1903, the international nineteenth- and twentieth-century items from Kunsthistorisches Museum's Stallburg gallery were returned to the Österreichische Galerie in 1986.

Under the Austrian Art Restitution Act of December 1998, 239 artworks in 76 dossiers have been established since early 1998 along with various supplementary dossiers. The provenance of the items in the museum library has also been comprehensively documented. The Art Restitution Advisory Board has recommended the restitution in forty-five cases of sixty-nine works of art and six printed works.  Fifty-nine of these objects from thirty-five collections have been returned to the legal successors, the remainder kept in the museum until all of the heirs have been identified. Prominent cases involving the works by Klimt, Schiele and Edvard Munch from the collections of Alma Mahler-Werfel and the Bloch-Bauer, Lederer, Rieger, Zuckerkandl families have attracted particular media attention. In 2006, for example an arbitration court decided that five Klimt paintings, including both portraits of Adele Bloch-Bauer, should be returned to the Bloch-Bauer heirs. The fortunate repurchase of several restituted works such as Hans Makart's Entry of Charles V into Antwerp from the former collection of Valerie Karplus-Lieben has meant that these paintings are now legally in the possession of the Österreichische Galerie Belvedere.