Shortly after the situation with Mr. Milodragovich (which I wrote about a number of years ago), Viktor Navorski shows up at Frank Dixon's office and tries to give him a Virgin Isle sailfish for his wall, apparently trying to mollify Dixon's anger. Navorski must have remembered the other fish displayed on Dixon's wall, visible in a scene earlier in the movie when the two first meet:
The fish that Navorski offers, however, is considerably larger than Dixon's fish (briefly glimpsed again in this scene as the camera pans with Dixon as he sits down), and this is significant.
Immediately after the problem with Mr. Milodragovich, Dixon had threatened Navorski as a punishment for cheating the system and allowing Mr. Milodragovich to leave with the medicines for his father but without the necessary documentation. "Do you think this is a game? Do you think I need an excuse to put you back in that cell, to keep you there for another five years? You go to war with me, and you go to war with the United States of America." Dixon made these threats in front of the review board that was there to inspect the airport and assess him for the role of chief field commissioner, a position he's wanted for years, and now he blames Navorski for making him look bad.
In this particular scene, he repeats some of his threats: "Let me make you a promise, Viktor, and this is a promise that comes from a man who is stuck here, a man who may be stuck here for the next ten years. From now on, you and I are partners. If I stay, you stay. You will not set one foot in New York City, not a single toe in the United States of America. Do you understand what I am saying to you?"
Navorski answers, "Yes, you don't like [the] fish."
It's sort of a humorous response, but it can also be taken in a somewhat more literal sense. Dixon doesn't like the fish that Navorski freely offers because it's bigger than any of his own and makes him look bad in comparison, just as he did in front of the review board.
