The institutional roots of the Heeresgeschichtliches Museum (Museum of Military History) go back to the eighteenth century, when a public exhibition of imperial armour, ceremonial weapons, war trophies and mementos was shown in the imperial armoury from 1759 in Renngasse, Vienna. It moved in 1856 to the Arsenal, built in 1849, where the k. k. Hofwaffenmuseum was opened in 1869 and the reorganized contents of the former arsenal were shown. In 1881, Emperor Franz Joseph ordered this court weapon collection to be incorporated in the k. k. Kunsthistorisches Hofmuseum, and it was transferred in 1888 to its new home on the Ring. In 1885, the Emperor approved the founding of the k. k. Heeresmuseum proposed by Artillery Inspector General Archduke Wilhelm and promoted in particular by Crown Prince Rudolf as chairman of the board of trustees. The k. u. k. Heeresmuseum, as it was now called, was opened by the Emperor on 25 May 1891. It evolved slowly but steadily until the First World War. The long-standing director Wilhelm John (1903–1934) and his deputy Alfred Mell were instrumental in defining the museum's orientation. In 1909 they curated the Archduke Karl exhibition in the k. k. Österreichisches Museum für Kunst und Industrie (Imperial Royal Austrian Museum of Art and Industry), which was extremely successful, attracting around 265,000 visitors in two months. On 27 July 1914 – the day before war was declared – the Heeresmuseum was closed to the public and was not to open again until September 1921. Wilhelm John and Alfred Mell continued to collect systematically during the hostilities in order to provide a comprehensive presentation of the war. The previous collection and exhibition focus was considerably enlarged and military structures behind the lines, the hinterland and the socioeconomic and cultural implications were also highlighted. John also managed through contacts with k. u. k. Kriegspressequartier (war press office) to obtain an extensive collection of war pictures, amounting by the end of the war to around 6,000 items. Although it was closed to the public, it lent objects for numerous exhibitions and was largely involved in particular in the two war exhibitions in the Prater in Vienna in 1916 and 1917. The end of the war put into question the future of the museum, which belonged to the Royal and Imperial Army, which was about to be disbanded. It was not until March 1920 that the Republic of Austria put the museum, now called the Österreichisches Heeresmuseum, officially under the responsibility of the Republic's military authorities. In spite of the initial intention of breaking up or selling the museum's inventory, it ultimately remained intact for the most part, with only the Hungarian government receiving a larger number of items until 1932. During the following years, the museum was involved above all in the consolidation and processing of the large collection of items from the First World War. In 1923 it opened the first new permanent exhibition area consisting of a war picture gallery with artworks from the time of the First World War. Other new sections of the permanent exhibition devoted to the war were not opened until Alfred Mell (1934–1949) became director, taking over after John's sudden death in 1934. These new exhibition areas brought important innovations in terms of presentation and the first extensive use of the medium of photography. At the same time it aligned itself with the historical policy of the Austrofascist corporate state, which sought an ideological link with the Habsburg monarchy. On this basis, Mell succeeded, despite fierce opposition from other museums, in being allocated space on the first floor of the middle tract of the Neue Burg, in which a large permanent exhibition on the First World War was to be installed.

The annexation in March 1938 and the new history policy put an end to Mell's plan, despite the fact that he had euphorically welcomed the annexation to Nazi Germany and was also willing to assist in the comprehensive reinterpretation of Austrian military history to reflect the Greater German and Nazi ideology, for example by installing a bust of Hitler and other First World War generals in the Hall of Fame, the most imposing room in the museum. In 1938, the army high command put the museum under the responsibility of General Friedrich Roese, the Berlin-based Head of the Military Museums, thereby considerably reducing its autonomy. After the Second World War broke out, the Heeresmuseum was used increasingly for Nazi propaganda purposes and for the ideological mobilization of the population and members of the Wehrmacht. The presentations of Austrian military history were replaced by "spoils of war" shows and special exhibitions on the course of the war. Some of these were organized by other Wehrmacht departments but most were done by the museum staff. The propaganda purpose of the museum assigned to it by the Wehrmacht meant that more shows than ever before were presented, its budget doubled between 1938 and 1944, greater interest was taken in it by the media, and visitor records were broken. On the museum staff in 1945 were four applicants for NSDAP membership and two members of the NSDAP. Of them, only Ludwig Jedlicka had joined the Party before 1938. None of them had leading functions between 1938 and 1945. All six were dismissed in 1945 against Mell's wishes. Mell had been able to retain his position as director in 1938 without joining the Party. The art historian Bruno Wikingen, who was responsible for the museum's art collection, clearly sympathized with the Nazi ideology without ever joining the NSDAP. In 1919 and 1920 he had belonged to the Sturmkompanie of the Freikorps Marine-Brigade Ehrhardt and had been involved with it in the Kapp putsch in Germany on 13 March 1920. The only person to suffer from discrimination by the Nazis was section and later department head Eduard Wettendorfer. He was initially classed as a "first-degree Mischling" and threatened with dismissal from October 1938. Following Mell's intervention, he was allowed to stay provisionally although he was not able to assume a leading function. After a decision by the Reichssippenamt (Reich Genealogy Office) in October 1942 confirmed that he was not a "first-degree Mischling", he became head of the 4th department. As a military institution, the Heeresmuseum was forbidden from 1938 from making purchases from antiques dealers deemed to be Jewish. At the same time, it attempted actively to profit from Aryanizations and the seizure of collections, reporting its interest in doing so to the Zentralstelle für Denkmalschutz (Central Monument Protection Office). Over fifty objects were acquired by the Heeresmuseum in this way, along with fifty-eight assigned to it by the Gestapo. It also bought some artworks and other objects directly from persons persecuted by the Nazi regime. According to our present knowledge, they made up only a small proportion of the over 24,000 objects acquired between 1938 and 1945, consisting above all of allocations from the Heereszeugämter (army ordnance depots) and to a lesser extent of articles brought back by Mell from his travels, most of which are no longer in the museum collection today. Because the museum belonged to the Wehrmacht, salvaging operations did not begin until autumn 1942. They lasted until autumn 1944 and were managed by the director and senior air raid warden Adolf Spalenka and Eduard Wettendorfer. The salvaged objects were distributed among fifteen depots in Vienna, Lower Austria, Upper Austria, Styria, Burgenland and the present-day Czech Republic, which made it more difficult to guard and conserve them. On 10 September 1944 an air raid destroyed the north-east wing of the museum building, and further offices, depots and museum workshops were damaged in air raids on 11 December 1944, 15 January 1945 and 23 March 1945.

In June 1945 a Soviet trophy battalion removed around 16,000 objects, including practically the entire small arms collection and some of the swords. Some of the depots were looted in subsequent years by the local population and occupying troops. Around 40 per cent of the entire inventory is thought to have been lost, with up to 95 per cent in certain sections, such as the model collection, the collection of modern uniforms, and the rifles. In January 1946 the Federal Ministry of Education (BMU) took over responsibility for the Heeresmuseum, now called the Heeresgeschichtliches Museum (HGM). Alfred Mell remained its director until 1949. He was succeeded in 1950 temporarily by Géza Kövess and ultimately, until 1956, by Rudolf Pühringer. Although a subject of controversy initially, the BMU had the destroyed part of the building rebuilt. Between 1945 and 1955, the HGM acquired around 20,000 new objects to fill the gaps in the collection and financed its own reconstruction through the sale of gunmetal. When it reopened on 24 June 1955, for the first time in its history the entire area of the building was available to it, and the new exhibitions covered the political, historical and cultural aspects of all military epochs. From 1957 there was a room dedicated for the first time to naval history, although the display ended with the year 1914. It was not until 1988 that the permanent exhibitions also included the First World War. The years between 1918 and 1945 were added in 1998.

After 1945, the HGM restituted twenty-one objects from collections expropriated during the Nazi period to their rightful owners. A further seventeen objects that should have been returned under the restitution laws had been lost through bombing or looting. Since March 1998 the Heeresgeschichtliches Museum/Militärhistorisches Institut (HGM/MHI) has also been involved in the work of the Commission for Provenance Research and has compiled complete or partial dossiers and reports on Oscar Bondy, Viktor Ephrussi, Eduard Epstein, Hanns Fischl, Siegfried Fuchs, Emilio Hofmannsthal, Saul Juer, Albert Klein, Gisela and Ernst Pollack, Albert Pollak, Geza Rado, Alphonse Rothschild, Franz Ruhmann and Valentine Springer.  The Art Restitution Advisory Board recommended the restitution of two hundred nineteen objects, while the search for heirs has not yet been completed for seventy-two objects.